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ABSTRACT . • 

The Program Standards Evaluation System was developed 
in response to evaluation requirements in the 1973 ilehabilitation 
Act, The system ift^ludes procedures for using' standards data to 
monitor* and evaluate vocational rehabilitation/ (VR) service outcomes 
and outputs as well as standards on key prbcedural .issues. This 
report contains the outlines of tHe analytic paradigm' for management 
'use of. the Program Standards Evaluation System; tjia,t is, the typical 
ways in. which information obtained through the operation of the 
standards syst&m can be an^ilyzed and acted upon by VR program 
managements In/the seven chapters of the report, the analytic 
pStradigm preseiits the f ollqvfing: ( 1) the program evaluation standards, 
and associs^ted data elements^^indicators of .success in achl^ing VR 
pliicement ^oals; (2) the relations among the program standar^Ss; (3) 
the'options 'available for setting expectations on those indicators, 
and a recommended process; (4) the decision support system, for 
investigating the bauses of problematic attainment and for ' 
identifying corrective actions; (5) the , system: for reporting 
achievement and for identifying and explbring problematic attainment; 
and (6) program managers' use of the components of the Program 
Standards Evaluation System. Appendix A (bourid separately as CE 0.36 
210) includes the detailed decision support tables and displays for 
standards data elements, while Appendix B reviews a number of . 
alternative approaches to. die setting 6f performance levels. (KC) 



*************************************** 

* Reproduction^ supplied by EDRS ar^ the best that can te made * 

* from the original document*. * 
************** ***^*****************************************^* *********** 



I 

VOCATIONAL REHABILITATION PROGRAM 
STANDARDS EVALUATION SYSTEM 
FINAL REPORT .. 



VOLUME II: US^G THE SYSTEM: 
AN ANALYTIC* PARADIGM 
FOR MANAGEMENT 



June 11, 1982 



\}S. DEPARTMENT OF EDUCATION 
WlONAL INSTITUTE OF EDUCATION 
EQUCATIONAL RESOURCES INFORMATION 

. CENTER (ERIC) 

V This document has b«en rtproduccd as 
recen/cd from the person or orgamzatwn 
origtnattng it 

Mmof changes have been made ^ improve 
reproduction qualtty 
Vp^t7Sv«w or opinKjns stated m th» docu> 
ment do not neccssarrfy represent oHiCtal NIE 
portion or poTicv 



"PERMISSION TO REPROOqCE THIS 
MATERIAL HAS BEEN GRANTED BY 

TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)/' 



Prepared for: 

/ 

Rehabilitation * Services Administration 
Department of Education 



Prepared by: - 

Berkeley Planning Associates 

3200 Adeline Street 
'Berkeley, California 94703 
(415) 652-0999 



00- 



MliUiiiiittMiiililttiiliililiMiia 



BPA project staff: 

Contract HEW 105-79-4000 ' \ , /. 

Susan Stoddard, Ph.D., Project Director ' 
. J. Mark Rogers, Deputy Project Director for Program Standards ^ 
Linda Toms Barker, Deputy Project Director fo^v Project Standards 
Richard Dodson, Ph.D. , Project Co-Director, 1979-80 

, 4 

Staff: Sherry Almand, Joanne Bain, -Liaida Barrett, Dena Belzer, 
Eileen ^Bleeker-, Frederick C, Collignon, Ph,D., Deborah 
Daro, i^oine Giadman, Keith Gregory, Patricia Jenny,'. 
Fran'Katsuranis, Ph.D., Shirley Langlois ,- Joe Mancini, 
Carol Meek, Judith L. Rill, Victor Russell aifd^Taleb' 
Whitaker 

Contract HEW 105-76-4116 

"Frederick C. Collignon, Ph.D., Principal Investigator 
Susan Shea, Project Director and Principal Investigator 
Linda Barrett, Project Director • ) 

Staff :^ Gail Chiarrello, Bevefly DeGraaf, Richard Dodson, 

^^^harles Froland, J. Mark Rogers, Gloria Root, Bruce 

Schmidt, Susan Stoddard, Chris Thompson, Debprah 
Wilkerson 



Authors of this report were: . * • \ ^ 

, Ri'chard Dodson, Susan Stoddard and J. Mark Rogers, with assistance 
. from Caleb IVhitaker. Appendix A -(trees) was prepared by Susan * . 
Stoddard and^J. Mark Rogers, with the assistance of Shirley Langloi? 
and Patricia Jenny of BPA and iJr.'Paul Mueller of the ^Xlalifomia 
Department of Rehabilitation.' 



This project has been funded at least in part with federal funds from the 
U.S. Departmeff^ of Education under Contract 'Number HEW-lOS-79-4000. The' 
program standards which were tested and refined in this contract were 
developed in a previous BPA contract, HEl^-l05-76-4116: "^The contents of 
this publication do not necessarily reflect the views orpoljLcies of the 
U.S. Department of Education, nor does mention of trade names, .commercial 
products, br organizations imply endorsement by the Uf^S. Government. 



• / 



9 \ 



I 



/ 

JABIE OF CONTENTS 



.1. INTRODUCTiaN ......... ^ .... , - . . . . 1 

The Program St^ndar'ds^^valua<:ion System ^ .^^ . 3 

, The Qoafs of the VR Program. 5 

The Analytic Paradigm. . . . , ^ 

^ . \ ' , • 

Organization of This Report ^ 6 

II. THfe PROGRAM STANDARDS. AND DATA ELEMENTS . 9 

Performance Stanflards ' 

• ' ' ' - 

^ Procedural. Standards I . : . ' 

» * ' _ _ 

III. RELATIONS iMONG THE PROGRAM STANDARDS. . - - - :_ - • 

Relations Among Measures of Perform^ance . . . 50 

^ Approaching the Problems ' 

I . / 

rv. SETTING VR AGENCY OBJECTIVES Via-A-/VlS 1WE STANDARDS 61 

^Should There Be Objectives?. . . 61 

/ E.xisting Approaches to Setting Qbjectives ' ^"v 

A iVev^ Proposalcfor Setting VR Agency Objectives.^ . . . \ . . . 64 



I 



J - 



V, L-iVESTIGAT-ING' PROBLEiMATIC ATTAI{)!MENT:" DATA-BASED DECISIpN 

/ . ,/ SUPPORT SySTEM 67 

The Process of Problem Identification.N • 68 

Examples . . y v. . 75 

Evaluation Research.^ ! ' ^ . 87 

VI. THE VR PROGRAM STANDARDS EVALUATION SYSTEM REPORTS 

Vri. PROGRAiM MANAGEMENT ANt) THE USE OF THE STANDARDS SYSTEM.. . ^ . . 107 

A Review of th-e Pr/gram Managers *Use of the Standards 

' System .../..... v . . 107 

Actors and Corrective AcJtions ^ ...... . lOS 

. devaluating the' Wan^lards System Over Time .109 

Chan^zing the St^-yndardiS -System . - 10^ 

Implementing th6 Standards System. . , . ;. 110 



TABLE OF CONTENTS (cont.) 



REFERENCES . : : IH 

APPENDIX A: The Rehabilitation Executive's E^valuation ^ystem ^TREES) 

t (Bound separateliT^' 

APPENDIX B: Dimensions of a Method for Setting Objective^ , — ...... . . B-1 



N 



ERIC 



LIST OF TABLES' , 

" 1. VR Program Standards and Data Element?: Final Reconnne;idations . • 10 
/ 2. Tie^ 'Between 'Standards and .Goals of.VR ... \. ...... i. - i*' 

3. Cbrreiation Among Measixres — Knuce, Millar and Cope. .... 

y 4.* Factor Analj^sis of Output Measures' Dodson (1978) . 

5. Problems -With A Conqjosite Score: ' Hypothetical Attainment' on\ 
Two Data Elements i 

6. ' Problem: Expenditures Per Rehabilitation Are Too High: . . ^ . 

7. Expenditure per Rehabilitatipfi * . 7S 

• 8. Expenditure^^er Closure and Expenditure per Case * 77 

a. Post-Acceptance Clostire Ratei. . ; i. /;....".. 78 

10. ^ Acceptance' Rate ' 

^11. , VR Cl^ljs Per 100^000 State Population .U. ^ 79 

• 12.. Percentage of Clients in Status 06.. I 80 

Percentage of - Ineligible Applicants . ^ . . ^ . . . : i .) . . • • 81 



13. 



S3 

'57 

74 



14. Applicants As a Percentage pf State Population. . . . 

15. Categories of Non-Competitive Placements As a Percentagje of 
Total Placement . 



'16., Coapaarisons of Work Status Olyectives and Outcomes, 
17. Achievement on Performances Standards. 



81 

83 
-84^ 



18. Achievement on PefformaAce Standards : ^ State Coraparisor. 

* * 

19. Achievement ^ on Performance Standards: Sub-State Con^aap^ons. 
IQJ Achievement in Perfprmtoce Standards: All Agekdes . 



. 94 

4 

. 96 
98 

. 1 •.• . 99 



« • • ; ^ \ . .. 

LIST OF FIGURES 

The .'^gram Staijdards. Evaluation System . . . 

Model Case Flagging System \ ...... i .... . 

The Flow (Jf the Declsion-Suppon: System 

Investigating a Problear Expen^tures/Rehabilitation ^ 

Too Hi/gh^ . . • : : . . . . j,. 

Exasrole Displays o£ Steps in Analysis o£ Rehabilitation 
Problems. . *. • 



4 



r t . 

I 



% 



9 



4 



} 



■ f 



My/ * 



7 



r.^ iNTRODUCTION * 

« 4 

The 1973 Rehabilitation Act contained, among its^^many <^her provi- 
sions, a requirement ^that evaluation standards., be devised and implem^ted 
to measTjre the performance of the VR program, in achieving its mandate. . 
Over the last four year^, Berkeley Plaiming Associates (BPA] , under con-' 
tract to the Rehabilitation Slices Administration (RSAJ, has developed 
an integrated standards evaluation system* Two distinct sub-systems ^were 
(^eveloped*' One, the proposed Program Standards Evaluation .System, evaluates 
the' federal-state VR pro^pms. The other, the proposed Project Planning 
^B^d Evaluation System, measures the effectiveness of indivichial projects 
and program authorities, funded by RSA discretionary funds. The ^nnipose^ 
Program 'Evaluation Standards include eight Performance Stiaxdar^ and 
associated -data elements and five Procedural Standards and associated 
data e^elaents* The Per fo rm a nce Standards pertain to service outcomes 
Cpro<h2Ctivity, effi^ivaness impact) , wh^le the Procedural Stand«ds per- 
tain to service method and process *Ce»g. , case handling, data quality). 

' During the Izst two year? , parts . Af the standards syst'ea were pretested 
in six state VR agency model ef^aluation units (MEl^s) ~" in the Oregon', Penn- 
sylvania, Delaware, and Virginia, combined agencies, and in the' Mississippi 
jblind agency. The primary emphasis in f his pretest of the Program Evalu- 
ation*' Standards was on the new data collection instruments and on revisions 
to the stanciards and associatl^data elements. ^ 

• The program istandards system^ is a system for ^evaluating and managing 
parts of the VR system, .This report is the second of four -volumes of ' ^ 
the Final R^ort on the Vocational Rehabilitation Program Standards 
Evaluation System. Volume I reports the findings of the pretest.. Volume ^ 
III, the Guidance Materials ,, contains detailed instructions on the stand- 
ards 4nd* their data elements,-- including special information requirements 
and forms.. Volume IV, Training Materials ♦ is designed as a series of 
training modules on the. material contained in Volumes I - III. This 
report. Volume. II, outlines the uses .of the standards and^ presents the 
standards Analytic Paradi^, showing how the standards supifer^ state , agency 



decision-making and point to specific actions for program improvement or 

changeJ . * 

♦ /* 

The purposes of the program standards system ar?, simply: 

\ ^ • . 

• to make available information on the achievement of state 

VR agencies with respects t^ VR goals as nyasured by the ■ 
^standards data elements; and, more .importantly, . 

# to guide -the behavior of state VR agencies toward greater 
adhieveom#'on those standards^ data elements; as well as 

^ to identify possible problems and corrective actions, when- 
ever s^l^te VR agencies are unable to reach, their achievement 

objectives. . , \ ' 

» 

The revised standards system thus sKaxes with the currentf standards 
system the purpose of providing information to RSA, to the state VR agen- 
cies, and to other; interested parties such as 0MB and Congress on tie 
achievement of the state VR agencies. Current achievwnent and Jiistorical . 
statistics will be provided in the VR program as a whoi^ and on. e^ch ' 
state VR agency. ^• 

• What is new about' the revised standards system is that it is oriented 
to guiding the behavior of the state VR agencies in*new directions, not 
jusir reporting on* p.ast behavior. The revised standards system-oS pros- 
pective, not retrospective, oriented to suggesting diredtions for ,futxare 
behavior and not just to teporting on past behavio^. By setting obj^ecti^ves 
. for each state VR agency to achieve 09 each of the standards* data elem^fev^s , 
thi VR system can be guided in the directions' that RSA and the states want 
to go^ • The overall direction of the VR program thiis can be changed, as 
can the achievement of particular state VR agencies. ^ 

Thej)aradigm of t^ie- system is concerned withflagsfMg problematic 
attainment, invesi;igating possible ^^roblems> and identifying corrective 
actions. • * * 

What is also very unique .about the reviseti standards system is that 
it'does not stdp when a state VR agency fails to meet its objective on a 
particular stiandard data element. Instead, in* the* revised standard/s system 
the decision support system ^identifies possible .problems and corrective * 
^actions. . This /System is designed to enable program. managers to quickly 



identify wither possible problems can be identified or whether further 
evaluation is required. 

TOE PROGRAM STAN DARDS EVALUATION SYSTSM 

The Program Standards Evaluation System has several components, as 
shown in Figure 1: * 

# Standards and Data Elements • A set of eight -Performance 

■ ' i 

* Standards and five Procedural Standards, with associated 

( data elements, measure the rehabilitation goals and functions 

o^^^the VR progajam^ with respect to coverage, placement rate, 
cost -effectiveness, impact' of client services, compliance, 
data quantity, and the proces^^ of service* delivery. . 

# Process for Setting Objectives > A process for setting 
objectives for each state VR^ agency on each of ti^|f standards 

# . data elements provides clear expectations for achievempnt, 

' expectations thaf are set in conjunction ''with each agency; ^ 

#1 Reporting System . A reporting system presents the levels . _ , _ 
of achievement of state VR agencies on the measures of the 
goals and function of the VR system which are captured in 
the standards data^ elements* Information on past acjiieve- 
jnent, the achievemeAt of other state VR agencies, the com- 
ponents .of the^ data elements ^^.'^d on informational data 

• elements are also presented. 

# Day>-based Decision Support- System . Possible reasons for prob- f 
lemat^c attainment of a pairticular *state VR agency on a partic* ^* 

^ular data- element are identified, either through investigation 
by program managers or through further evaluation research* 
In addition, corrective actions are identified fbr-each pes- 

• sible problem* 

As- can be seen from Figure 1, all four of these components are oriented 
to the management of the federal -st^e VR program and to the measurebent 
of achievement of VR goals • 
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Figure 1' 

The Prograa Standard Evaliiation System 



/ Standards and Data 
Elements Measuring 
> Goals and* Functions 
of VR Program " 




Process for Setting 
Ob.jectives vi5-a-vis 
the Standards 




decision Support 
System to Identify 
Probl^ns and 




THE GOALS OF THE VR PROGRAM 

The standards were developed after a careful investigation of all the 
possible candidate areas o^^^ VR client service program. RSA has issued 
a variety of goa-ls over the course of its existence. These goals haVe 
appeared, in such publications as the RSA Forward Plan, short** arid long* 
range plans. Information Memoranda, and so on. The current^ goials of the 
RSA program stress the importance of job development, and job placement. 
The p rogr a m standards wiH improve the management informatidn ava?.lable 
on placement, since many of the data elements measxxre specific dimensions' 
of the quality of client placements Ce,g., competitive employment, wage - 
level). The VR program's coverage of clients eligible for -derives, and 
the effectiveness and efficiency of the program in pl^x^i^S^^^^^^^f 
meflaiingful jobs is the focus- of the p rogr a m standards. The procedural 
standards add some additional coinpliance measures.. 

One point to note concerns the potential users of the standards sys- 
tem. The goals mentioned above were set out. by the Commissioner as pri* 
orities for the, federal rehabilitation agencyl One might mistakenly aSsxme 
'that Jincethe^ standards operationalize ^thosi^ go^ls, the standards informa- 
tion system is of 'use only to ^ISA. To speak plainly^ nothing could be ^ 
further frtm the truth. First, since the basic VR program is a, state-federal 
partnership, the goals set out for the federal agency by the Commissioner 
apply to the rehabilitation pfograms operated by states. Second, it is 
neithe* intended nor ^desired that states be frozen out of the processes, 
embodied, within the standards system, nor that the isystem should have 
little relevance or use' by states in their management processes. In fact, 
the opposite is true. We hope that the" standards system will provide 
states 'with performance measures which contribute to their management 
capabilities by. defining the objectives of a quality program. ^In addition, 
the standards system should useful to state managers, sinc.e performance 
expectations can be applied, to sub-estate units (districts^ counselors. 



1 ' * . . 

Besides Job development and JobPlacraent, the other priority areas 

are: Internal Management , Facilities , I1VIU>/IEP Linkages , Deaf -Blind 

Program , and Learning Disabled . 



b.ffices) by ^t^te agency .managers. This/.of couirsej, wouJLd be at the opticyn 
of 'tiie^ states- Ot|iir cdyrelW state- VR agency- s]^fexfic goa^ and func- 
tionst-^will .also continue to eici'st and'to also guide the behavior of the • 
st;ate VR agency. 

THE'ANALYrrC PARADIQl . • ' . . ' 



..' Webster's Dictionary- defines the word Vparadigm" as ^'an outstandingly 
^clear^of typical examplTe or aj^etype." Our pt|rpose in this report is t^o ^ - 
provide just such 'an example .r- ah ^xanple of a. specific set of activities 
which form.the compiet^u^ystem^* Specifically in the pagejs which follow- ^ 
^e_set_out the broaE outlinSs^of the analytic payadigm f0r manaji^ement use 
o£ the ProgrA Standards'' Evaluation Sy?teitt.. That is, we djsa^s the "typ icaj,^^ 
waysr. in which iaf omit ion oHtained, through the operation of the standards 
system can be analyzed and acted upon by VR p r o gr am management* The stan^' 
dar^s system iftseif^ presisnts the means td an end: throu^ iiiqjlemen^ation ^ 
of the stahdaxds system we enhance the '*manageafiility" of the program. The 
^d goal is to improve p rogram performance- throu^ the means of enhanced 
manageability. The purpose of the analytic paradigm is to guid^ the man- ^ • 
ageaedt use of the pro gr a m standards system, to tie the information to - - , 
cotrective actions, to planning and to policy-makinff.\The standards alone 
will not serve the pxirpose of improving per fo r man ce; it is through the use 
of these measures in progr am maii^<g^ent that the .goals of the program can 
be achieved. ' 

ORGANIZATION OP THIS RE?OIgr 

Our purpose hiwre, then,, is td lay tAjjipgroundwork for a. comprehensive 
system for analyzing informatd^dn. obtained from the standards system,, and 
for acting on that information to direct the program in the achievement 
of VR goals. In oijder to do this, the analytic paradigm presents: 

e the Program Evaluation Staiidards and*, associated data ' $ - ^ 
. .elements — indicators of 'success* in aftthieving VR - 

placement goals; . - 

e the relations among the Program: Standards ; * ; • ^ 

• the options available for setting expectations on those ^ 
indicators J and a recommended 'process; 
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# 'the- decision support syst§pi for investigating the causes 

of problemitic attainment and for*identi^dji| corrective ; j 

actions r • / * * X 
'0 the systejri fqr^ reporting achie^emeirt/and for ictentifyin£ ^• 

, , and exploring problematic attainment; and • . . 

p program managers' of* the component*^ of the Program;j , 

Standards, Evaluation System. • 

^In Ghapter'"^!; we discuss the ^proposed standards and; data elements, 
and their, relataon to the goals and funqtions' of the VR program • Both the 
•performancfe Standards and the Procedural Standards are discussed. 

because "of tha large number of dfeasures of the VR progr^ contained - 
^ in the- standards 'data elements. Chapter III is concerned with the relations 
"between the sta^iards data, elements and with the problems those relations v 

* cause • Firsr, ]±e" empirical relations among several measures, of perfor- 
mance are examined* ThW, some possible approaches to the problems caused 
by the relations among the standaards are. presented* Finally, the impli^^ 
catibiis of continuii^g wit^ multipie measures are pointed out. 

In Chapter IV, tfhe process for-setl?ing objectives vis^a-vis the 
r 'Standards* is discussed. Having j<i«xtified and op.erationalized the measures 
,o£ the goals' and fuactign* of the VR^4J5j>^raa, we need a systeiy for setting 

* objectives which tak'efk into.acxMunt such tiungs as the conflicts among 
goal^i the levels of .attainment whi'ch are reasonable to expect, given the 
ctnrrent resoixrces, practices, priorities, and technologies; and methodo- 
logical soundness of measures of effectiveness. Fir^t^, the question of 
whe^er theire. should be objectives is raised. Three existing methods for 
setting objectives are. described, and a new proposal for setting objectives 

' is put forth.. , 

In Chapter V, we discuss in detail the profcedures which should be , 
undertaljen to identic the causes *of problanatic attainment and ciorrective 
actions. This conqjonent is called the data-based decision sxq^port system. 
This system provides RSA and the staije VR agencies 'with an approach to 
investijatinff the causes of misatisfactory attkinmjeftt and .for idenilfifying ^ 
« correctivL actions . This chapter jfis organized as follows : a^ general 
iBodel of the thinking process that maangers would undertake to investigate 
the causes of problematic attainment ii presented fir^t, and two examples 

* 

, ,, Oi 14- ' i ■ • 



of how the thiiiking process ,woiild be applied to data on the revised stan- 
^ daxds are given next. Finally, there is a review of techniques that 
might* be employed in those^cases where additional research is needed. 

Chapter^ VI explains the reporting system for the standards. The 
information to be contained in ^he system is identified, as are the Jdnds 
of reports and displays to be generated. , 

Chapter VII presents an overall of 'the management iise of the Program 
Standards Evaluation System- and sotne, suggestions for ^implementation, 
* Appendix A includes the detailed decision sr^ort Tables .and Displays 
for standards data elements • Appendix B reviews a number of alternative 
a^proachejS' to the setting of performance levels • * / - 



' . u . . . . 

II. ' THE PROGRAM STANDARDS AND DATA ELEMENTS 

4 This ch^ter introduces two types of standards, which malje up tne 
p rogram standards system: There are eight perforaancr standards, which . 
inclXKie; data 'elements to measure, specif ic aspects of agency performance. 
^ITiese .are numerical measures <?f aspects of agency coverage, efficiency, 
and prograa impact In addition, there are five procedural standaards ' 
coveadMj areas of program con^^liance with regulations 'which are iatpozrtant 
to as^fc service quality, Hrhe revised standards md^their data elements 
are shown in Table 1* 

^The- development- of these, revised program standards 'aiid associated data 
Weoents was the restilt of a contract with the Rehabilitation Services 
Adminxs^tion (RSA)* "This development was {he result of a technical and ' 
political process, which included: - 

^ reviewing 'previous woiic on standards for the VR program, 
incluMing the- New Orleans Report, and the work of the Urban 
Inst]itute('''aad JVfK', Inc^; 

revfeiHjxg the existing, standards, those published in the ^ 
Fed^raJL Reigister December 19, 197S; 

gathering' reactions and recommendations 'of -^selected state 
agencies; 

; # depicting various conceptual approaches to standards devel- 
opnen^ and' of various criteria for standards development; 

e listing ^candidate areas for the selection of standards,- along ^ 
with recommendations for .revised standards, usin^ the concep- 
tual approaches and criteria for justification; 

e choosing data elenents to assqciate wi,th each standard based 
on the criteria developed; 

•>, involving RSA staff and members of the special CSAVR advisory 
committee in each- stage of the development; and 

^» pretesting the standards and their data requirements in six 
Model Evaluation Units (MEUs) . • 

- i' / ' - • / 
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Table 1 



. VR PROGRAM STANDARDS AND DATA ELEMENTS: . FINAL RECOHMENDA^IONS ' 



PERFORMANCE STANDARDS AND DATA ELEMENTS y . ^_ 

1. Coverage . / \ 'I ' \ 

VR shall ^serye the maximuo proportion of the potentially eligible target 
pbpiilation, subject to the level of federal program funding and priorities 
r among clients, \ ^ \ ^ 

(i) Clients served per 100^00(7 pdpulation • ' • 

Cii) Percent severely disabled served • 

2, ■ Cost-Effectiveness, and B^efit-Cost Return. 



the VR prtjgram s^iall use resource's in a ^ost-effective manner and shov^ a * 
positive r etu r n to society. oJ investment in^cational rehabilitation of 
disabled clients, . ' ' ' » ' . - ' 

(iO Expenditures per competitively employed ciosxire . 

(ii) ^Expenditoire per 26 closiire • . * 

fiii} Ratio pf total VR benefits to total V5 cost« (benefit -cost ratio) 
(iv) I\ Total net benefit frcm: VR services (discounted^ net p»s4ht value) 



3, RehStffiiitation Rate ' , . 

VR shall maximize the number and proportion of clients accepted for services- 
who are successfully rehabilitated, subject to the pufeting of other statdards* 

(i) "Percent 26 clostares ' ' ^ * 

^tii) * ^ Annual change in number of 26 closxrres ^ " ' . , 

"Economic Independence , . ' 



Rehabilitated clients shall evidence economic independence. 

(i) Percent 26 closures with weekly earnings at/above federal mijjimum wage 

(ii) Comparison of earnings with competitively employe^ 26 closures tp 
earnings of employees in state 

S. Gainful Activity . ' ^ 

There shall be maximum placement of rehabilitated clieints into competitive 
employment* Noncompetitive closures shall represent an improvement in 
gainful activity for the client, • * . \ . ^^'i 

/ (i) I>,ercent 26 closures competitively employed ' ' " ^ • • 

(ii) pUcent competitively employed 26 clostires with hourly^ ewnings at/abov^ 
federal minimum wage 1 , 

^ * 

(iii) Percent noncompetitive ly employed 26 closures* showing improvement in ^ 
function and life status (implement after FAI/LSI pretest) 



J Table i (continued^ 
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6, Client Change / ^ . ^ 

RehSbiiitated clients shall tvidence vocational gains. 

(i) Comparison or earnings fajeforQ and after-^VR'^ services ' v 

r \\x) (In addition, changes in other statuses/, and functioning; ability, 
when such measures become available-) ^ \ 

?• A ^Retention * - . ^ ' , / ^ 

Rehabilitated clients^ shall i;^etaip^r^Ile benefits of..VR services. 

(i) Percent 7n rTnTijjrni 'rtlTlTiriiTiff esurnings at follow-i:^ 

(ii) Comparison of 26 closures with public assistance as primary source 
of ST ip p ort > at closxire^d follow-up. 

(iii) Percent, noncompetitively employed 26 closures retaining closttrei 
skills at follov»-up^ (implement after FAI/LSI pretest). 

8 ♦ Satisfaction 

Clients shall^e satisfied with the VR pr o g ram, and rehabilitated clients 
, shall ap pr a ise VR services as useful in achieving*" a^ maintaining their 
•vocational^ objectives* " ' 

(i)^ Percent closed clients satisfied with overall VR^ experience' - 

(iij' Percent closed clients satisfied with: counselor, physical . 
•restorjLtion,^ job training services, placement services 

Ciii) ifercent 26 dlosxjres judging services received^as useful in ' 

^ obtaining tJxeir job/homemaker situation or in c ur re nt perfSraance 



< 
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PROCCTUKAL STANDARDS ' -t , ^ 

, 9, Tt«^00 Validity ^ ^' * • i • . 

Information collected on clients by the R-300' a™i all data reporting 
systems used by RSA shill be valid, reliable, icfcxirate, and complete • 

10, Eligibility ^ l ^ - > ' - ^ . 

*^ • Eligibility decis-ions shall be based on accurate and sufficient dia^ostic 
information, and VR shall continually review and evaluate eligibility 
decisions to ensiire that decisions are being made in accordancp^with laws 
and ^eg]udation5. 

11* Timeliness 

VR shall ensure that eligibility decisions and client mbvemeht through the . 
VR process occur s a tiinely^ manner appropriate to the needs, and capabilities 
. of the^ clients, 

12, iwRP ' : 



• ' VR shall provide an Individualized Vfritten Rehabilitation Program ^r each 
applicable client and VR and the client shall be accountable to eaclt ^other * 
for complying with this agreement. , , 

13,. Goal Planning ^ ^ ' 

Counselors shall make an effort to set realistic goals for clients. Compre- 
^ hensive consideration must be giveh to all factors in developing appropriate 
vocational goals 'such that there is a maximum of correspondence between gpals 
and outcomes: competitive goals should have competitive otxtcoaes and non- 
competitive- goals should have noncompetitive outcomes/^- - 



/ 
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The ^relationships of specific standards to VR program goali are shovm 
in Table 2* The system piirposely does not inolude measures of inputs (e^g*, 
what kind of VR counselors are hired, what kinds of-*^ervices are' prescribed) , 
operation of related agency sxibsystems (facilities, CAP^, mobility training), 
or me^isxxr^s of financial^ operations (budgetary systems, financial management 
information systeias) • Thus the performance and procedural standards ' data 
elements are mostly oriented to ^measmring the performance of.VR agencies 
♦in helping individual cliapts; they^ leave the decisipn of how to achieve 
these performance goals to individual agencies and VR qgUnselofs* ^ 

In this chapter, we will f^t review the performance standards and 
thtfir data elements. For each of the standards, the data elements will 
be defined and discuss^. Following the performance standards, the five 
procedural standards are introduced and described^. 



PERFORMANCE STPfmA&DS 



Below,, each of the eight performance standards is discussed as to. 



. rationale « The section defines each data element used: as admeasure of 
a standard, and describes the coiqponents of each measure (In a later 
section, pages 3S-4^ , each of the procedural standards will 'be-* reviewed- 
as well.) • ^ . 



Pretest volume al'so reviews outside conmenta on thp standards 
and data elements, as well as responses to those comments. Mor^ discussion , 
of the" weaknesses of each standard and the data element is also Cont^ned. 
Also,f there is an analysis of the data from the- six MEUs., The Guidance 
volume summjvrizes the final 'recommended form and data sources for the system. 
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Tabtle 2 

Between Standards and Goals VR* 



Goals 



of VR 



Standard 



Coverage 



Efficiency 



* QMality 
and Iropact 



Co apfla nce 



Data 
Quality 



Process 



Perfoirmance 

r. Coverage 

2. Cost Effec- 
tiveness and 
Benefit/Cost 

3/ Rehabilitation 
Rate 

4. Economic* 
Independence 

5* Gainful • 
Activity. 

6* Client Change 

7, Retention 

Satisfaction 

procedural 



V 
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X 
X 
X 
X 



9. Da>^ Validity 
and iteliability 

10. Eligibility 
Determination 

U. Timeliness 

12. IWRP 

13« Goal Planning 
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STANDARD I: . V OCATIONAL REHABILITATION SHALL SERVE THg MAXIMUM PROPOR- 
, TION OF THE POTENTIALLY ELIgl&LE TARGET POPULATION , 'SUBJECT . 

TO THE 'level OF FEDERAL PROGRAM FUNDING ANP- PRIORITIES AMONG 

♦ ' CLIENTS . . : " > 

Data Elements : (i) 'Clients served per 100,0(30 population^ 
(ii) Percent severely disabled served ■ 

Thi5 szsaidzrd addresses coverage, or- the extent to which the voca- 
tional rehabilitation p r ^ gir a p is serving the eligible target population. 
The neei^ to ensure accessibility of services to all the eligible disabled- 
is of paraaoimt inportance to RS<I and the states. Given this standard's ^ 
focus, we'feei thit it fits in well with the costreffectiveness goal. On 
its face,. the standard is concerned with the "effectiveness" aspects of 
the cost-effectiveness quesrions: increased service covetagi of the eli- 
gible population is one indication of increased effectiveness on the part 
of a state agenpy. In ^^tct^ coverage represents .one of the most Ipasic 
aspects of a pr o gra m's effectiveness Alone, thxs standard ignores 
considerations "of the quality of the coverage *(i*e., the appropriateness 
and utility of the pro^pram's ^^ivities in the clients* behalf, and the - 

clients' service outcoine;5) . ' HoWever, these considerations are addressed 

h 

by other standarxis. " ^ 

DATA ELBOT 1 Ci) : CLIENTS SERVED PER 100,000 POPULATION ^ 

Although this data eleaent does not provide a^true estimate of the 
level of coverage of e-ligible target population, it does, however, pro-- 
vide, a projty measure of the size of the target population by using the 
overall state population. If estimating the targW population. were a 
straightforward matter, this proxy would not be desirable or needed, but 
given the need for long-term development of an acceptable target popula- 
tion measure, this proxy serves a potentially useful purpose. Also, it 

* > -mm 

is used now by state agencies, and thus it has some management utility 

and validity as a performance measure. The form for the elanent is as ^ 

fbllows; 



y 



» served. in a gjiven year 
state population Cm 100 ,000 's) 



er|c ' C'. 
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flATA ELEMENT 1 (ii) ; PERCENT SEVERELY DISARLED SERVED 

• # 

The proportion of severe disabled vdtHin a caseload can reason- 
ably be expected to impact negatively on a ^state agency's total volume 
(i.e., caseload size) and on its costs. With. a high tumovet^ of severely 
disab/ed clients., time in process would-be expected to increase and com- 
selor capacity decrease, thus decreasing a program^ caselpad volume poten- 
.tial;' that is, a decrease in coverage* To effectively assess .co;verage, 
the* proportion of the caseload that ^is severely ULsablei^ must be taken 
into account. Further, given the legislative importance attached to ser- 
- vice to s^erely disa&led, it is most a]^ropriate to include this data, 
element under the standard on coverage of the eligible ^client population. 



* severely disabled served in a given year 
total f served in a given year 
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STANDARD 2: TOE VOCATIONAL REHABILITATtON PROGRAM' SHALL USE RESOURCES 
• IN A COST-EFFECTIVE MANNER AND SHOW A POSITIVE RETURN TO 
SOCIETY OF INVESTMENT IN VOCATIONAL REHABILITATION OF 
DISABLED CLIENTS . ^ ^ 

Dati Elements : (i) Expenditure per competitively .employed 26 closure 
CU) . Expenditure per 26 closure 

(iii) Ratio of total VR benefits to total VR costs (benefit.- 
cost ratio) 

(iV) Total net benefit from VR services (discounted net 
present value) 

This standard is the one most: obviously related to the program's 
cost-eff*ctiveness goal- TWo issues are addressed by this standard. The 
first i:f*^the issue of cost •effectiveness: with the financial resources 
avaittflue to the state (dt siib-region, or district, .or counselor) ^ how 
successfully did it achieve desired objectives? The second issue revolves 
arotmd cost-benefit concerns (i.e., •'return on investment"} • Specifically, 
the standard asks the question: Are we- gercirtg more out of the program 
than we put ia7 C ur re n tly, benefits from the program are measured 
primarily ia monetary terms ^ terms o!^;^ges earned, taxes- paid, 

and public assistance J^aregone). Unfortunately; this. "haxd-nosed«» monetary 
focus ojjdts consideration of 'many of the other benefits .derived from VR 
(e.g», increased functional capacity). Work is well underway to develop 
methods for taking such benefits into account.^ In the interim r however, 
monetary cost-benefit measures will- continue to be important, particularly 
in the current era of budget constraints and intensifying scrutiny of 
governmental activities. As such, RSA can profit from use of a cost -benefit 
measure in terms o£ public relations value, as well as in terms of improving 
its own self-devaluation capacity. 

Many ^lifferent cost-effectiveness data elements were considered in the 
design of the Standards. Essentially, any data element requires in the 
denominator a measxare of program achievement and in the numerator some 
measure of resources of the kind which the agency is particularly anxious 



1 ^ ' • . 

This work has been recently completed by the Texas Institute, for' 
Rehabilitation Researdh (TIRR) . The TIRR Final Report entitled "A Benefit- 
Cost Model fpr the State/Federal Rehabilitation Program" is available from 
RSA. * - " ' * 
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to use efficiently. We chose total expenditures for the ,nximerator because 
it overcomes, various accounting problems, for example, "what is a Service 
cost" ancf ^Vhat is a counselor PTE?" The denominators were chosen 
because of their relatively universal acceptance as measures of "success/' 
Benefit *cos^ models estimate total benefits and total costs- in terms . 
of dallars* These models 'are neutral with regard to type of delivery ✓ 
strategy. As such they do not penalize agencies whick ch^e to spend 
more per client in ord«r to produce better results • Any cost-effectiveness 
measTire, on the other hand, focuses on rewarding states which minimize. costs 
in achieving a given objective • Units of the outcome measiare are asstaaed to 
be equal in value, i«e., one rehabilitation (26 closxxre)^ is as ^good as any 
other rehabilitation* In order to offset these limitations, it is required 
thar the outcome or benefit measure be one which decision-makers are pre- 
pared to view as having high value and units of equal value* 

DATA ELEMENT 2 Ci) : EXPENDITURE PER COMPEnTIVELY EMPLOYED 26 aOSURE 

^ This data element compares total agency expenditures to the number of 
competitively employed 26 closures* It applies the most: stringent criteria 
to the measiarement of costr-effectiveness.by JSocusinig on only those -26- 
closures^ who are competitively employed. Such a priorityrmay not in fact - 
be desired, particularly given the recent, emphasis on service to the 
severely disabled* However, we included this data element because we feel 
that, historically and even today, a consensus exists that competitive 
employment: is one of the higher quality and most desirable types of closure 
obt^ainable* The form for this element is as follows: 

total agency expenditures ^ 

# competitively employed 26 closures 



DATA ELEMENT 2 (ii) ; EXPENDITURE PER 26 CLOSURE ^ 

- litis cost*e£fectiveness measure ^relaxes the measurement^ criterdCa 
somewhat to allow ''credit** for all types of rehabilitations «. It recognizes 
that some clients are not capable of achieving competitive employment and 
that other employment outcomes can represent achievement comraensxirate with ^ 

— — ■ ■-■ — — — . . ..... ..^....^^-.......^ — ...^.^f.^^.^^^.^..^^.u.^.....^i^ 
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a client's abilities. This data element compares. total agency expenditures 
to all 26 clostires, thus captures the effect of gainful activity, whether 
it lies in the realm of tompet^ve or non-competitive employment. The 
form for thi-s element is as follows: 



total agency expenditure 
• ' # 26 closures 



DATA ELBOTS 2 CUil AND 2 (iv) ; 

(iii) RATtO OF TOTAL VR BENEFITS TO TOTAL VR COSTS ' (BENEFIT-COST RATIO) 

(iv) NET TOTAL BENEFIT feoM VR SERVICES tDISCO^NTED.^fET PRESENT VALUE) 

These- two data^ elements axe very^ similar in concept; therefore, the^y 
will be discussed -together. The forms of these data elements are shown 
below. " " ' 

Data Element 2 (iii) • 



ERIC 



benefits 
costs 



Data Element 2 (iv) : 



Beixe^^cost modeling of social service delivery systems enjoys cur- 
rent vfide acceptance as a measuremei^t tool.' Its. use extends cpnsiderably 
beyond the VR field. The figures provided Ipy benefit-cost aaaly^ yield . 
a single number,, whidi is an immediate indicator of program success .• 
Beckiise of its surface simplicity, and because it is «a popular sophisti- 
cated analytic tool for evalttat;ing prlgra^p wortijr'the benefit -cost, of the 
VR system is included as a standard. 

As a review, for the National Science, Foundation has not. ed, benefit-'^ 
cost applicat^Lons in th* VIT fie Id 'are more extensive and have generally been 

more sophisticated (or at le«t at a higher level of technical quality)^ 

' • 1 w 

than in most other social service and manpower prograa areas. There are 

^Berkowitz and Anderson. 7PA0EC An Evaluation of an Experimental 
Rehabilitation .firo ject ^ ..Rutgefrs University 1974 . \ 
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Benefits Costs 
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a numbdr of- models available for use. In one case,' RSA commissioned the 

development* of a model for routine use by the program, which was designed 
to be adaptable to the needs of many xxsers (i.e,, state agencies, RSA con* 
tracted evaluation studies, RSA 'itself) and to be capabi# of periodic up- 
dating and refinement as new data became available. That model, developed 
at the University of California, Berkeley and sixbsequently refined by BPA 
staff, has been u^^ed by RSA, severaj state agencies, the Urban Institute, 
Abt Associates, National Analysts, and Greenleigh Associates, among others, 
usually under RSA regomaiendation This model is the basis for the two 
data elements proposed for use 4^ measuring the costs: 

^ ' (B enefits) 
a. Benefit -cost ratio ^ — costs ) 

' b. Discounted net present value '(Benefits-Costs) 

Both of these formulae use the''!social discounted" present values of - 
benefits and costs, and both' use the same components to^arrive at benefits 
and at costs- These components, in brief, are as fbllows: 

Bcrpefits ^ 

^ discounted value of paid earninjgs; f . 

» change in output of homemaker clostires; 

e change in output of unpaid family workers; ' 

e change in ^'after hours work" (e.g., homemaking tasks per- 

fanned by wage^aming rehafailitants) ; ^ 

, e fringe benefits;. • ' ^ 

^ change in output of families of rehabilitants (as a result 
of rehabilitants assuming homeinaker tasks) ; 

# redu^ons in public assistance benefits; ' * 

e repeater costs' (a "negative! beneftt") . ^ ' 



^Frederick C. 'Collignon ani Richard Oodson, Benefit-Cost Analysis of 
Vocational RehabiUtation Services Provided to Individua ls Most Sevetely 
Handicapped (ISMH) April 1975. . • 
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Costs 




total program costs during the fiscal year, minus carry- 
over costs and maintenance -costs; 

e costs borne by parties other than VR; ^ 

e research, ^traificing, and danonstration costs; 

e beneSts. foregone by clients during participation in VR 

services (i.e., anV wage and ftringe benefits foregone by 
clients with earnings at .referral) ; and 

^ dient-bome costs for VR services* 

The ratio . (B/C) provides a .measure of the relative value of benefits 
to costs. This measure standardizes this comparison, and can be used for 
comparing" values across programs, states, or sub-state areas. The benefit - 
cost ratio is often used irL legislative reporting. B/C can also be used 
to observe -change over time within a single agency. Our understanding of 
progra m gain is increased by looking not only at the relative magnitujles 
of benefits and costs, but at their absolute difference as well. . 

The net benefit measure (i.e*, B-C) is the preferred approach- of 
economists/ It is very sensitive to the scale of program operation: in 
the caSe of VR, for example, larger agencies would produce, greater total 
net benefits than small agencies, simply because of their lars^rj^eloads* 
Thus the measure is in a ppropr i ate for comparing across' s,tate agencies, but 
is useful for observing change over time within an agency. 
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STANDARD 3: VR SHAa^MAXIMIZE TOE NUMBER AND PROPORTION OF CLIENTS 

MlCtVtEd FOR SERVICES WHO ARE SUCCESSFULLY REHABILITATiED, 

' SUBJECT TO TOE MEETING OF OTOER STANDARDS , 

Data Elements : (i)' Percent 26 closures « 

(ii) . Annual change in number of 26 closures 

, Traditionally, success in VR has been measured by the nuniber of ''26 
dostxres," or stxccessful rehabilitations obtained^ The VR goal is to 
rehabilitate clients, and to ignore that goal in the standards systen 
would be' a serious and uncomfortable omission, VR does need to know how 
many individuals it successfully serves and must have encouragement to 
rehabilitate as many persons in need as possible, 

"DAtA ELEMBiT '5 : PERCENT'WCLDSURES . 

This data element provides^ a straightforward measure of an agency's 
success in rehabilitating the clients 'it accepts for services. The data- 
element, focuses, on the proportion of clients accepifed- for service Ci-*'» 
excluding "D8'$)' Who jure successfully rehabilitated. i 

- » of 2d closurtes 
♦ # of 26+23+30 closures 



DAfA ELEMENT 3 lix) : ANNUAL CHANGE IN KUMBER OP 26 CLOSURES 

' »» " 
This, data element attea^ts to assess an agency's success in maximizing 

the number o£ clients accepted for services who are swcessfully. rehabilitated 

TTie measure uses the state agency's prior performance as a baselinej for 

determining success. in "maximization": thaj^is, an agency is judgeSd to 

have maxidzed the number of rehabilitants if it has increased the number ^ 

of 26 closures by some previously specifiLed amount* That >pount will have 

been set by the state agency, ^ ♦ , ' 

(# of. 26 closures in' current year) - 
C# of 2(6 closures in previotis* year) 
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STANDARD 4: RSiABILITATED CLIENTS SHALL EVIDENCE EC ONOMIC INDEPENDENCE 

Data Elements : (i) , Percent 26? closures with weekly earnings at or 
' above federal minimum wage. 

(ii) Comparison of earnings of competitive employed. 
► 26 closures to earnings of employees in state 

VR's most basic purpose is to assist disabled persons in finding' 
gainful eapXoyment. The extent to ^hidx cUents improve ;:^eir ability to 
be economically self-sufficient (i-e*^, "independent") tbrou^ gainful 
employment is a -fundamental, concern of VR* - 

Aghi waamt of economic independence is one fecet or closure quality 
"of inte^st to VR, and thus-' this s tand a r d is included in that group of 
standards concerned with the qjiality of services; however, Veconomic inde 
pendence" means ..different things depending on the type of 26 closure 
obtained* Thus, a. variety of data, elements are needed to capture the 
concept* 

In addressing the- measurement of increase in ecxjnomic independence f 
clients rehabilitated, the logical place to look is to waae? and wage 
increases • Two data eleaents are used to assess wages-: -.-comparison to 
.the national standard (the minimum wage) and comparison to state norms* 

D ATA ELEt^ 4 (1]: PERCENT OF 26 aOSURES WITO WEEKLY EARNINGS^ AT OR 

: ' ABOVE THE FEDERAL MINIMUM WAGE • \ 

When attempting the measurement of economic independence for rehabil- 
itated clients, particularly in competitive employment, the logical place 
to .'look is to wages. The ftrst data element for this standaid compares 
the wages of wage-earning rehabilitants. to the "standard" of the federal 
minimum wage ; There are state minimum wages that may be higher than the 
federal wage, and not all employers must pay federal minimum wages under 
all circumstances. The Jiormative implications of this data element are 
that a disabled person should be expected, under equivalent circumstances. 
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to make at least , the jninimum required by law for citizens of the U.S. 
The form for thii data element is as follows: 



# 26 closures with weekly 
earning level at or above 
federal minimum wage 
# 26 closures 



pat;? ELBIENT 4 (iil : CaMPARISOy OF EARNINGS OF COMPETiTVELy EMPLOYED 



26 CLOSURES TO EARNINGS OF EMPLOYERS IN STATE 



In this data eftment, the wages of rehabilitants axe compared to a 
standard or wage* rate for the general population, as in (i) • In this 
instance, however, the focus of the comparison i§ mean wage of closures 
with wages to the mean wage of ^employees with wages in the state-. 

This iaethoji controls ftar state*to*state variation' in earnings levels, 
whereas using the federal minimum wage as a denominator does not* Other- 
wise, |he concdpt behind this data element is the same as with- (i) : -to 
coiflpai?e the-wftgei of rehabilitated disabled clients to- those of the '^general'* 
- population • In %qpa» respects, this *is a more comprehensive indicator than - 
datii- element (i), because it provides an estimate of clients' "standard of^^ 
living*' relative to other persons in the state ♦ In general, .a^s wage levels - 
increase, so does the cost of living, and the amount of income required to - 
maintain an "acceptable" standard of living. Since the data.element iiicor- 
"porates cost of living (via sta:te wage norms), we obtain a better measure 
of clients ' living .standards relative to the surrounding environment* In- 
contrast, the "federal mtri™"" wage is not set with' reference to local cost 
of living. consideratibimK. The {am is as folloWs:^ - , * 



Mean weekly earnings of competitively s 
. employed 26 closures , 
Mean weekly earnings of employees in state- 
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STANDARD S: THERE SHALL BE MAXIMUM PLACEMENT OF REHABILITATED CLIENTS 

INTO COMPETITIVE EMPLOYMENT. NON-COMPETITIVE CLOSURES SHALL 
REPRESENT AN IMPROVEMENT IN GAINFUL ACTIVITY FOB THE CLIENT . 

Data Elements; (i) Percent 26 closures competitively employed 

(ii) Percent competitively employed 26 ^closures with .. 
hourly earning at or. above the federal minimum 
wage 

' (iii) Percent non-competitively employed 26 closures-^ - 
showing improvement in function and life status 

» * * 

Like Standard 4, this standard concerns the quality of .clostires. ob- f 
tained by VR agencies . Historically, .competitive' OTptOym'ent hais be^n se'en 
as the best kind of "tlostire* However, competitive -enqj.l'oyis.ent: may not Be 
the- appropriate placement for all clients. Still, VR reg^lations require 
that any placement of a successfully closed client be into "gainful and 
suitable employnent:,"^ "consistent with his/her^ capacities,"^ whether in 
competitive, sheltered, or non-competitive employment^. 

Given thwe reqxxirements we have tried to incorporate, several con- 
cepts into' Standard S. Before discussing the data elements ia'detail, we 
present an. overview .of the concepts, the general focus of whid)L is on 
gainful employment; thus, tiie data elements measure the foUowirjg:;: 

(1) the extent of competitive employment closures, since com- 

petitiv? employment ^still can be -seen as the best' type or , • 
closure;, 

(21 the extent to which competitively employed 26 ^closures eaafn ^ 
the Warly minlmTP w^e,. as an indicator of minimum standaids 
for gainful employment; and * 

C3} the exterit to which non-competitive closures have obtained some 
benefit from participation in VR. *; . y 

To reiterate, this standard is concerned with the quality of clostire, 
as- evidenced by increases in gainful activity whether. of a paying nature 
or not. Competitive employment is seen as the highest- form of gainful 

^Rehabilitation Act of 1973, Section 302 Cb) C2) CB) • 

. Vedcral Register, "Implementation Provisions," 1361, ICbb), Novem- 
ber 2S, 1974^ 
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activity; however, allowance is made for the need to make some non-competitive 
:ci(JS\ir»sr Still, in those cases, VR^is to ensure that the benefits in terms 
of ""gMn£dT activity - were obtained, even if not of a vocational nature. 

DATA ELEMENT 5 (11 : . PERCENT 26. CLOSURES COMPETITIVELY yPy^gP ' 

For a standard emphasizing maximum placement: into competitive employ- 
ment, perhaps the most obvious data element is tp count how many are so 
placed* This data element is a simple, straightforward measure of degree 
of success in placing closures in ^competitive employment and could be 
easily implemented as the data are readily and currently available from 
the R-300. The form is as follows: - 



t competitively employed 26 closxires 
# 26 clostsres 



DATA. ELEM ENT 5 Cii3 : PERCENT CQMPETITIVELY EMPLOYED 26 CLOSURES WITH 
: / — riOURLY EARNINGS AT OR ABOVE FEDERAL MINIMUM WAGE 

" This data elaaent applies -more stringent criteria 'to the measurement 
of '^'aarunua placement of Trehafailitated clients into conpetitive enployaent."' 
It- "compares the number of 26 "closures with hourly eMnings- at- or above the- . 
federal tn-^^-?iwtm wage to the total nunbeij of 26 crosurea* As xn data element 
4(i), this "data element implies that a disabled person in the competitive 
laSor market should be expected to earn at least the federal minimum ^wage* 
^\3alIk^'4^iL) however, this measure ^represents an ea^loyee's worth to the 
employer • Total weekljr earnings are an indication of an ea^Jloyee^s 
financial well-being, while his/her 'Vorth" may be deternaned by examining 
his/her hourly wage* Thus, this) data element provides a measiure of the ' 
'♦value'* of rehabilitated VR clients who are in the competitive labor 
market relative to the- federal minimum waje*- The form, is as follows: * 

#26 cldsure's with hourly earnings 
' > at or above federal minimum wage .__L 
if 26 closures 
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DATA ELEMENT 5 (iii) • PERCENT NON-COMPETITIVELY EMPLOYED 26 CLOSURES 

• SHOWING IMPROVEMENT IN FUNCTION AND LIFE STATUS 

> T [ ' 

As Stated earlier, closures into non-competitive employment may be 
legitimate for certain clients- Nonetheless, if VR is to claim any credit 
for '♦rehabilitating** clients into non^competitive employment, then there 
must be some injiication that VR helped improve those clients' capacity for 
gainful activity. If the client obtained no benefits whatsoever from VR, 
then VR has essentially wasted money and time- Obviously, such outcoiws 
axe not desiarable*^ ' ^ - 

^ This data element takes a subje^ctive approach to the problem o£ 
assessing the legitijikcy and appropriateness of non-competitive closures. 
It is computed by ^taking the percent of non-competitively employed 26-closures 
who state they have done any of the following: iM^TOved their self -care 
abilities axi4 thus freed other family mtabm to join the labor force; ex- 
perienced improvement in any self-care or homemaker-related func^onsi 
experienced improvement ia job-related skills; or had "improvements '♦ in, 
attitude.. This wide range, of indicators messes the extent to which non- 
competitive closures benefit as a result of 'intervention-. The form, for . 
tljis data, eleipent is as follows: 



# non-competitive 26* s with improvement 
on LSI -FAX measures from plan to 'closure 
# ncmcoaqpetitive 26 's 
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JSTANDARD 6: REHABILiTATED CLIENTS SHALL EVIDENCE' VOCATIONAL GAINS . • 

Data Elements: (1) Average earnings -change of 26 closures., Ijefore 

versus after VR services 

. (2) Other changes^ functional ability and life status 

, It is axiomatic that, after VR services, rehab i lit at e'dS tt ents should 
evidence some sort of vocational gains; either in monetary or nonnnonetary 
terms. This standard assures that attention will be paid by the,VR field 
to. the level of client changes • It supplement? the concern for measu2;ing 
post-se3rrice outcomes (as in Standards 3-5) by using the client's pre- 
service circumstances as a baseline for comparison. • . 

J)ATA,ELB!ENT 6 Ci) : 'AVERAGE EARNINGS CHANGE OF 26 CLOSURES, BEFORE VERSUS 

AFTER VR SERVICES 

This data element is included because wages are the mos,t straightforward 
indicator- of vocational change. Weekly earnings are used to measu re chang e. 



[ -r^ : : 

(Sua of closure earnings for 26 closures) 
minus '(sum of referral eamipi^s for 26 closures) 
#26 closxires ' ~ 



DATA ELEMENT 6 (ii) : QiANGES IN FUNCTIONAL ABILITY (FAX) AND LIFE STATUS 
^ (LSI) INDICATORS [ ' ^" — 

In addition to vocational change (as measured by dati element ^^i)^ 
the VR frrogran also acts as a change agent in terms of non^vocational ' \^ 
aspects of a^ client^ s dife* As with the data elemmts associated with^ 
non-H:oflq}etitive eaployment closures (as in data element S(iii))j the method-- 
alogy for assessing non-vocational change needs development This develop*- 
meat should occur as an outgrowth of RSA's FAX/LSI pretest*^ ^ Until, such 
time as the measxires can be finalized, no data collection or reporting, will 
^e conducted for this data, element 



; , a 

A prei&est of these measures is being conducted, by counselors with * ^ 
1,300 clients in the California and Wisiionsin VR agencies. A jreport 
indicating the resulta of use of these measures ^t the time of case intake 
and IWRP development is available from RSA. It is entitled ''Functional 
Assessment in .VR Clients; A Pretests," 
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ST^^lDARD 7: REHABILITATED CLIENTS SHALL RETAIN THE BENEFITS OF VR 
' -SERV-I.gES. • . 

Data Element's; (i) Percent' 26 closures retaining earnings at follow^ 

■ : ■ up . • 

(ii5 Comparison o£ 26 closures with public assistance 
as primary source of support at closure and at 
» - . follow-up 

(iii5 Percent non-conpetitively employed 26 dosiires 
retaining closure skills at follow-up 

« 

Retention of benefits gained through vocational rehabilitation ser- 
vices is important to the rehabilitated client tod as a .measure of overall 
program effectiveness. Job losses following successful closure imply 
prograia failure and point to incongruence of program goali" vis-a-vis in- 
dividual client goals. Are we "rehabilitating" clients temporarily to 
meet program objectives,, then finding clients back where they started a 
few months later? This question has a .great degree- of. importance to the 
overall VR mission and thus a standard in this area. is. highly appropriate. 
Aside from employment peases of benefit retention, ^^tional attention, 
is given to expanding iflie data elements foT ti^j^^t^t^^ to include non- 
enplbyment measures. - .v';'-*--'*.-".:'.'.' 



This standard embodies another., of thie cohcii^- related ,to quality ser 
vice" in the VR program. In the ideal setting, successful! Ly closed clients 
would be permanently rehabilitatetj^ The theory,' of coursu, is that if^ 
(Quality' services are provided, clients will retail the-.ab|lity to function 
in a job and to compete in the labor market. The manifestation^ of those 
abilities are that clients do in fact retain their jobs, or some job. Of 
course, it is not always possible to retain jobs (or eatings levels), - ^ 
regardless of the quality of services provided by VR. Clients may sxiffer 
from unanticipated relapses or complications of their disabling handicaps , 
which can cause clients to lose their employment capabilities temporarily 
or permanently. Alternatively, clients can be laid off from j<Jbs. due to. 
macro-economic conditions, regardless of the quality of VR services. 
Nonetheless, 'in general we would expect clients to retain benefits. The 
standard is appropriate for inclusion in the overall standards system. . 

. ■ CL .37 ; . ' • 
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DATA' ELEMEOT.7 (i) ; PERCENT 26 CLOSURES RETAINING EARNINGS AT FOLLOW-UP 

Since the achievement of "gainful activ/ty" is the basic goal for 
the VR client, a simple measure of the retention of that benefit is the 
client's continued employment. However, this data element tightens the 
criterion to consider retention of the client's economic welfare, level at 
dosixre. The data for such a measure would be available through follow- 
up inquiry. A recommended follow-up survey designHaavailable in 
-Volua^-H;^ of this Final Report, the Program Standards Wdance Mat erials. 
The. form of data element 7(1) is shown below: , 



26 closures with earnings at\ closure. 
who retained or increased eamingsSat follow-up 
# 26 dosuzes with eamings at closure, 
surveyed aV^ollo*'"'^? 



DATA ELEMENT 7 (ii) 



COMPARISON OF 26 CLOSURES WITH PUBLIC ASSISTANCE 
AS PRIMARY SOURCE OF SUPPORT KT CLOSURg ANP AT ' 
FOLLOW-UP " 



" This data element would provide a needed dimension iir assessing bene-r 
fit retention for non-competitively as well as competitively placed success- 
fill closures. Here, benefits, are proxied by measuring the extent of the 
clienti'_use of public resources. .By fbcusing on the degree to which there 
is a reduced need for public assistance, an emphasis is given to the econ- 
omic self-sufficiency of the client in -terms of stability^ improvement. 
While this represents only one dimension of the possible benefits associated 
with successfijl closure (and one less sensitive to jhe complete range of 
effects of VR services) , it has a high degree of face vaAdity. as a measure 
of public resource burden. The form of data element 7(i^y is presented 
below. " : • 



s 



% 26 closures with public assistance 
as ' primary source of support at follow 

ouol " 



*46 closures with public assistance 
as primary spurce of support at clqsurt 
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DATA ELStENT 7(iii') : PERCENT MON-COMPETITIVELY EMPLOYED 26 CLOSURES RETAIN- 

ING CLOSURE SKILLS AT FOLLOW-UP \ 

This data eleraenr extends the concept of retention of benefits to 
non-competitive closures, ixi terras of the benefits assessed in. Standard S, 
data element (iv). 'The data element computes the percentage of all non- 
competitively employed clients who have maintained or improved their 
closure skills. at follow-up. The percentage is computed in terms of the 
number of clients who stated they had improved on any aspect. o£ self- 
care, attitudes, homemaker skills, job-related skills,' or had "improved 
so as to release other family members to join the labor force/' Like 
Standard S, data element* (iv) , the exact, methodology for deriving this 
data element's needed information in a consistent and reliable fashion 
is problematic. The state of the art is in exploratory stages and the 
special follow-up data co-llection required involves significant reswrces. 
While the data element is currently low in implementation feasibility, 
the measure has considerable value frpm a conceptual' perspective. The 
form of data elanent 7Ciii) is shown, below: 

/ 



* non-competitively deployed 26 closures 

retaining LST/FAI closure skills 

# non-competitively employed 26 closures 

surveyed at follow-up 
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STANDARDS: CLIENTS ^ALL BE SATISFIED WITH TOE VOCATIONAL REHABILITATION 
PROGRAM, AND REHABILITATED CLIENTS SHALL APPRAISE VOCATIONAL 
REHABILITATION- SERVICES AS USEFUL IN ACHIEVING AND MAINTAIN- 
ING THEIR VOCATIONAL OBJECTIVES. ' ^ 

Data, Elements ; Ci) Percent closed clients satisfied 

VR experience • • . 

(ii) Percent closed clients satisfied with specific 
aspects of VR ^ 

* Ciii) Percent 26 closures ijuding services received to 
^ have been usefufin obtaining their job/homemaker 
'Situation or in cuxrwit performance 

As an. indicator of consumer appraisal of services, the standard on 
qlient satisfaction with vocational rehabilitatioij services has consider- 
able merit ♦ '^ince client satisfaction polls usiiSHly offer high degrees ^ 
of support for the program, this standard is viewed as having distinct 
political value in lobbying for expanded financial support ac both the 
state and federal levels? * Complementing the political utility of a (satis- 
fciction measure is the inclusion o£ a client utility assessment in the 
standard;. The intent of this clause is to zssbss whether successftally 
closed clients rate the utility of VR services positively in termis p£ 
actually having contributed to their getting a job and: fUnctioning^ in itr- 

As- a substantive" qualification of the satisfaction standard, utility 

assessment offers a valuable entree for probing areas heeding program 
improvement and for ensuring* consumer involvement in improving the res- 
pon*iveness o£ VR services to client needs. A jaodel for satisfaction 
.sunoRya_is-includei i^i Volume III^ the Program Standards Guidance Materials . 



DATA 'ELEMENT 8 Cil ; PERCENT CLOSED CLIENTS SATTSFTED WITH OVEBALL. VR . 

EXPERIENCE ' . f ' ■ . ■ "■ 

Overall satisfa^on as a measure of program performance hap several 
advantagbi^: (1) the procedtcre is in place as a part of previotB reporting- 
requirements ;M2) developmental costs have already been incurred; C3) it 
constitutes a composite measure of client satisfaction that responds to 
legislative and constimer advocacy concerns; and W the data show some 
discrimination among closure statuses* The form of this data element is 
as 'follows: i ' , , . 
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* closed clients surveyed satisfied 
.with overall VR experience . 
• ^ closed clients surveyed 



_- DAIA ELEMENT 8 Cii) : PERCENT ctoSED CLIENTS SATISFIED WITH SPECIFIC 

' ASPECTS OF VR - ^ rr^^-v^- ■ ■ 

^^^^^^^^^^^^^^^^^^^^^^ * * 

This data elenent attenpts to gain a mojre .detailed* picture of client 
satisfacticm with specific key aspects o£ the overall VR process . In 
particxilar, the aspects isolated tor inquiry include questiong^ afaom: the 
client's counselor,. the physical restoration services received^, the job 
training services received, and the j ob " plaeemeht process . Cons'istent 
negative assessment in any one o£ theSe areas would be highly useful in- 
guiding state evaluations and providing substantive input to ptograonatic 
xjiipx' ovement s • _ ^ * 



# closed clients satisfied 

a. with their counselor 

# closed clients STiryeyed . 

' , ' * closed clients sJfcisfied , 

b. with physical restoration ^services. 

# closed clients surveyed 

#' closed clients satisfied 

c. * ' with job training services 
' ^ # ciosed .clients surveyed 

* '# closedrxlient^ satisfied 

d. with job placement servi4:es 

# closed clients surveyed 
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DATA ELEMENT 8 riii l : PERCENT 26 CLOSURES JUDGING SERVICES RECEtVED TO 



HAVE BE^ USEFUL IN OBTAINING TBEIR: JOE/HOHEMAKER 
'^ItliAtttiN OR IN dJRRgNT PERFORHANCE . 



Rehabilitated clients can make fairly objective "assessments o£ whether 
the- services they received were instrumental in securing their outcome sit- 
uations . Equally as important as VR services ' contribution to the attain- 
ment o£ the client's clpsti^e situation, is the usefu'lnes? o£ the skills 
obtained in aissisti^ clients to functiprf in these hew poisitions.. While 
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not unequivocably objective, the client's assessment of whether he 9r she 
uses these skills and/ox knowledge gained ftom VR services is the closest' 
approxim^iOT of the case. The form of da?a element 8(iii) is shown Wfely. 



# 26"d6siires judging Services Tcceivedr • 
" .to have been 'useful i?x obtaxftirig 
tlieir job/homemaker -situation or in current performance^ 
26 closures surve/yed , 
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PROCEDURAL STANDARDS ■ 

The Procedural Standards consist of five goal-statements for the VR 
program, pertaining te R-300 validity, compliance with key regulations, 
and certain aspects of case .handling. They are standards 9-13 in Table 1. 
The Rrocedural Standards are intended dfi a method pf ensuring attention 
to four critical process areas, and to data validity. It is intended for 

'states to use the Procedural Standards to benefit their program evaluation 
efforts and facilitate the improvement of services to clients. These pro- 
cedures will form the basis for agency decisions to make appropriate 

.changes in practices, where current processes are not in keeping with 
client interests and positive progpcam performance. ^ 

The recommendations for the Procedural Standards reflect the desire 
to allow maximim flexibiUty to states in the VR process, yet still ensure 
attention to the areas addressed, by the. Procedural Standards and provide 
sufficient data, ia these areas to allow for programwide analysis. Ideally, 
a uniform procedure would be followed by all stares for monitoring these 
process areas, even though states retain differences in the ways they 
organize and conduct case service delivery ► Indicators of compliance . 
with legal requirements, such as eligibiUty and IWRP, should be the same 
for all states,. i.e., the same questions should be asked' and the same 
stpmary data should be reported. ^ 

Jfiosz of the needs of the Procedural Standards are. best met through 
case review. Thus, we are recommending that a single .case review process 
be implemented to address the case review need? of all four of the Procedural 
Standards; We recommend that the Case Review Schedule (CSS), developed by 
the San Di-ego State RCEP IX, be used as the basic document for Procedural 
Standards data collection. The CRS was mandated- by RSA as the standard- 
ized instrument to be used by regional RSA offiLces whenever they conduct 

. case-reviews.''" For Procedural Standards 10 (eligibility) and 12 (IMP), 
BPA has selected, the CRS -.items which we consider .essential to adequately 
assess, compliance. These items make up the Modified CRS, which is con- 
sider^ly shorter than the full CRS. RSA could choose either the CRS 
of the MCRS as th*» instrument for collecting Procedural Standards data. 



•RSA Information Memorandum, October 17, 1980. 



i'. \ih±le the^CRS ii an appropriate vehicle for collecting compliance 
data, it lacks certain items needed to assess the validity of RrSOO data 
(Standard 3) or to assess. timeliness of case*service (Standard 11). For 
thes^e two standards, BPA developed separate instruments to con^lement the 
CRS. These instruments are included in Volume III of tlje ^inal Report, 
Program Standards Guidance ><aterials. - If the Procedural' 'Standards are ^ 
'iii5)lemented, these two instruments would be uicoxiporated direc^tly into the 
CRS to provide a unified datai;collec^on instnfument.^'^ v"^/. 

, Haying"^ describe tS^^ general thruist.of the Procedural' Standards and', 
the general process for collecting, the needed data^ \we turn next to a ^ 
discussion of the- individual:. standards, '^'^ • - 
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STANDARD 9: R-500 DATA VALIDITY AND RELIABILITY 

Infonnation collected on clients by the R-300 and all data . 
reporting systems used by RSA shall be v:alid, reliable, . - 
accurate, and complete. ' ! ] 

• The VR service delivery systems needs an objective data base from 
which to. measure performance. ' Yet inconsistencies and errors in raportinj 
currently exist among and within VR program data systems, Confusi^m or 
misxinderstanding over definitions exist ahd need to be minimized. . This 
Procedural Standard would ensure that .state agencies maintain acceptable 
levels of validity and reliability in reporting of R-300 and other data. 
This standard assumes states' attention to good data processing is pertin- 
ent to ail the standards. Urns, given the importance of jgtiable, valid, 
and accurate data on which to' base the program's evaluation capacity, we 
feel that this- Procedural Standard relates to all of the broad RSA^^ goals: 
compliance, quality, 'and cosx*ef£ectiyeness ♦ , . * / ^ 

Reliability, accuracy and coiiq)leteness of data sfaotild be checked- in 
several ways^ Wiile we would recommend validity studies on a periodic basis, 
and edit checks as a ^art of routine data processing,, this standard en- 
compasses a specific recommended procedure for states to follow to enisure 
the accuracy of data recorded xand submitted to RSA ^through the Rj^OO^ 
Primarily, the case review process should include an accurac/ check between 
the case folder information, the R-300 form itself and,* if , the state has- a - 
computer system, computer output listing of R-300 items selected for reviews" 
In paxticuUr, those R-300 data itemjs which are used in computing the stan- 
dard ^s data elements should be. subjected to' checks of accuracy 'and validity 
through case folder docxzmentation . 
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STANOARD^ 10: ELIGIBILITY DETESlMINATIOM . 

Eligibility decisions shall be based on accurate and suffi* 
cient dia^ostic Inforniacion.- and VR shall continually revlevr 
and evaluate eligibility decisions to ensure that decisions 
are beitig made in accordance '»dth laws and regulations. 

The determination of an applicant's qualifications for eligibility is 
a 'critical poiftt in the VR process for both the client ^and^ the agency. 
This standard seeks^ to protecjt client interest by requiring state agencies 
to install procedures for monitoring eligibility decisions in 4 sample of 
'cases and ensuring that the decisions are appropriate > in compliance with 
legal requirements, and supported hy[ the proper diagnostic information. . 
This standard pertains to two of RSA*s broad. goals. First, inasmuch a4* 
the eligibility determination process rests on a legal footing, the s^-zndsxd 
pertains to the goal' coiqpliance with the legislation* Second, we feel 
th2Z it pertains to the goal of cost •effectiveness, since it is a misuse of 
money to serve ineligible persons, particularl}^ if other, eligible clients 
are turned aw&y due to an incorrect determination of ineligibility. 

In establishing a procedural standard for- the review of eligibility 
determination , we are concerned with the a p pr opr i ateness of the decision and 
itf accordance i^ith laws and regulations . Me expect information* from this . 
review to address two faMts of this con-ccm: (1) that clients who a:e'nor 
eligible for VR. services not be accepted for services, and C^) that clients 
who are eligible are indeed accepted.. ' - * 

Ifhile monitoring and review of eligibility decisions by supetvisii^g ' 
counselors or managers will provide a check on that determination, states 
have varying- supervisoryN?tructures and roles*" and should be allowed to 
retain flexibility' in their monitoring practices. Although we support a * 
cross-check on eligibility decisions, we are not recommending its inclusion,; 
as a requirement, for this standard. The Case Review Schedule Serves as 
the data source, for this standard. 



STANDARD 11: TIMELINESS - . * . * ^ 

VR shall ensxire that eligilaility decisions and client 
movenent through the VR' process occur in a timely * 
manner appropriate to the needs and capabildties of 
the clients > 

. - This standard seek? to^avoid delays in the VR process that are likely 
^^0 impede or Hinder siicces«^ rehabilitation of the .clients Rather than 
sw^a performance standard using time-in-status to define 'todue delay," 
this Procedural Standard requires that each, state have a tionitiaring'pr * 
flagging mechanism for cases remaining in statuses over a given length of 
time, and a procedure to evalxiate th€ appropriateness of any case delay • 
Many of the state VR agencies already have variationis of such a' system in 
place/ 

This standard pertains to the RSA goal of providing quality case 
services, for two reasons. First, one aspect. of the quality of a client's 
service e:q)erxence is the speed with which his or her case is handled: 
did the client feel tfiat the cotmselor' "cared" about him (as evidenced 
by the fact that the counselor 'Iccpt on top of things" an4.'Tcept things 
moving along*') , or did the counselor seem to pttt IJui on a l^er priority? 
Tie client's perception of his. treatment' by VR can have an impact on his 
attitude toward VR and about the 'usefulness of participation in VR-. 
Second » research on su!Ccessful rehabilitation outcomes has suggested 
a relationship between timeliness and success (perhaps as a consequence 
of tlxe perceptions discussed above) *r 

• The issTxe of timely case movement, or '*undue delays" (as it is phrased 
^in the current standards) has been one of long discussion and controversy. 
While there is literature to s uppo r t the correspondence between certain 
times in process (partictilarly time to eligibility decision) and otitqome, 
there have alsa been, questions about interrater reliability in the area of 
judging timeliness of case movement through case review. Nevertheless, 
an overall review of timely case movement on a client -by-K:lient. basis . 
is best handled through case review-, if items can be identified which 
have good interrater reliability. 

Much effort has gone, into attempts to define, and establish standards 
fdr, timeliness of case service progress. As noted^ research on success- 
ful rehabilitation outcomes has st^orted the concern for timeliness in. 
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establishing a relationship between the time re<paired for zji eiigfailit^ 
decision and ultimate client outcome « Previous attem^s to monitoi 
timeliness of service provision by way of a standard on "undue delay'*^ have 
been hampered, however, by several problems. Vxt first is the definitional 
\and reliability problem. '•Undue delay** and its 'converse, "expeditious" or 
^"quick and efficient" case management, mean different things to different 
people. The current standards use the approach of arbitrary time periods 
to define "timely^' case movement: . eight* months has been- defined 'as the 
limit for timely eligibility decisions; 22 months for timely completion of 
the VR process. This approach has ^een widely and justifiably criticized 
for its .Jack of sensitivity to the legitimate differences in individual 
cases: a complex case, perhaps involving long-term educational services, 
might well require more^ than 22 months, without any delay. Where a ca^e 
has been sxxbject to a delay, the situation is further complicated by the 
differing implications of different causes for delay: lack of client res- 
ponsiveneis?, inattention or inefficiency on the part of the counselor or 
the .VR agency, 'and problems Wside of ^ (failure of a vendor to deliver^^ 
unavoidable 'waiting lists in training programs). Each imply .very differ- 
ent responsibility for time lapses and cannot equally- be* ascribed to VR 
agency "failure." ^ . 

1 Thus, use of "objective" measures of timeliness has suffered from arbi- 
trariness and frequent inappropriatisness of ^established time cut-offs for 
nany clients;. Other approaches to' objective measurement, such as recording 
planned, initiation and^ completion dates for eadi service, and monitoring 
compliance w^th the schedule, suffer from cumbersomeness ilt execution, 
the, other han|i, subjective jx«igments of timeliness have been vulnerable, to 
criticisms ogisnreliability in ajiplication, HoweverT ^ uareUability 
may welJL hav^ arisen due to the iacoxpor»tion of tvrq distinct con^e^ts 
intp'the previously used Vdue delaiy" judgments. This term, ;'undu« ^^^r" 
includes .concepts of both tim.e lapse and judgment of blame,.-culpability 
or unjustifiable time lapse. (The-word "delay" itself sometimes connotes . , 
wili^ftuiiess or negligence; and the modifier "mfdue" definitely implies 
-such-problens.) Case reviewers might well differ ih judgments as to the 
cause of a. -delay, and thus whether VR should be held acfcbttitablej. ahd for 
Ye-«son-,-*eviewersjmay differ, in 'their .classification of a case, 'one ^ 



citing an undlffe delay, anotfier seeing an unfortunate time lapse, but being 
-unwill'ing to label it an undue delay i£ client motivation or outside 

vendors played a role. 

In response to the problem of a dual focus in assessing timeliness, . 
a new timelihess assessment instrument has been developed which relies 
upon reviewer jildgment, Tmt which divides case assessments • of timeliness 
ii^to two segments: first, a notation of whetheif^ delay has- occurred in 
terms of .time' lapse between necessary activities in a case; and second., 
an assessment' of 'the reasons for the lapie. , The relevant questions are 
appended to the Case Review Schedule and concern critical "phases of case 
progress - eligibility determination, development of service plan, and 
servxce delivery an4 termination. In addition, the TMeliness Assessment 
instrument allows for notation of whethe?r a -case ,was handled with "undue ■ 
spee4:f*^at is, if the case moved too fast, in the reviewer's judgment,' ' 
given the circumstances of tl^e case. While undue speed may be a less 
pressing concern than- undue cl'elay, the issue did come up during the 
standards pretest, and the Timeliness Assessment instrument has been 
revised to address the is?ue. 

The Timeliness Assessment ifistxument can be used by states in^ conjunc 
pion with the case-flagging mechanism for open cases, required 'by Standard 
11. A mechanism' must, be set up by each state to flag each case. which has 
remained in a given status longer than a specified period of time. Review; 
of the' client's situation should then take place (in a format decided by 
the state) to determine if case movement is appropriate, ^ut no reporting 
to RSA would be required. 

As part of its project to -revise the VR Program Standards, 'Berkeley 
•planning Associates (BPA) developed a model for -improvement , of case- 
'f lagging practice and use of time in status standards. at the state level* 
A state should -not flag too many cases, because such flagging would be 
inefficient. However, flagging too few cases will possibly leaVe too • 
many untimely case* in the system without examination. The model, there-- 
fore, is based .on examining the number of cases being flagged with the^ 
states' -existiiig 'flagging standards, in relationship to an analysis of the 

1 

service process and the overall caseload, and refining state flagging 
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standards.^' The analysis uses data from the 11-300 file (time in status) 
and timeliness data collected through the procedural standards instruments. 

Figure 2 shows the steps to follow in the analysis, of uniue delay 
in the caseload. A (Tase is selected for review, following the sampling 
design used by the state agency for timeliness review (100* of all cases, 
or state random sanq>ling procedure) . State timeliness standards (allowed 
tines in process) will be used in this model. The model calls for in- 
wards or downwards adj listing of these times in prodjpss standards-^ (in- 
creases or decreases in the aaxiatn time allowed in each status) based 
on two additional standards for the caseload itself: 

1. Not more than 20* of the flagged cases should be timely. 
If more than 20% of the flagged cases are judged as timely 
when they are reviewed, the system is flagging cases un- 
necessarily, and the flagging standards should be less 
stringent (times allowed in the stattxses could be- in- 
creased) . / - 

2. Assuming the conditions in (1) above hold, at least S* 
but not. more than 10% of all cases should be flagged. \ 
If such excessive flagging occu*s, and the flagging 
represents cases judged untimely, then, there is a prob- 
lem with the service delivery system itself, and an . 
analysis of the process is' called for. If - less than 

S% of cases are flagged, the flagging system should: 
* * be more stringent (times allowed in statuses should be 
decreased) . : 

The model in Figure 4 uses both quantitative information on times 
in status and subjective information firom^ the Timeliness Assessment 
Instrument to decide about cases. ^ In the first st*ge, a case is selected 
for attention. If time in process is, all right ^ for the case, it is 



^e full analysdls leading to* the proposed model can be found in 
^PA^s report, fteHew of State VR Agency Procedures for Case Flagging 
knd Quality Assurance (September 198i>,> available ftrom RSA. ' 

^ ' '^th^i mbdel'ii, however, concerned only witlltime^i-in-status which 
are too long ^ It does ^not address '|rushedf' cases . Cases handled with 
**uf(^ speed*' are a s^arate issue requiring sp)scial state attention. 
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Figure 2 
Model Case Flagging System 
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returoed-to the file. If'time^in process , exceeds agency s 
.^qasct.is. flagged and reviewed as to the timeliness of the prbcess. ' As we 
have pointed outpwhdle there is a relationship between timeliness and 



time in process, it is not a one-to*one relationship, so* it 
that jnagged cases Will be judged timely. If so, they also 



is possible 
can "be rettimed 
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to the file. For both these grcfups returned to the file (labeled A and B 
on Figure 4), the number of such cases should be recorded. Likewise, > 
the number of untimely -cases flagged. should ]3e counted. (In the figure, 
this is CO Cases shoiild be flagged and reviewed until thej planned sample 
size (A.^* B ^ is achieved* 

Once the sample is complete, the system asks three qtulstions of the 
cases. Fiyst, do the timely cases (B) exceed 20% of all magged (B 
cases? I£ yes,, the system my be flagging too many, cases/ and times • 
allowed f(jr each status could be increased* If, however 7 B/CB + C]) is 
less than 20%, the systeja asks whether* less than 10% of ythe total case- 
load was flagged* If so, then the time in process staa!dard5 appears in . 
equilibrium for the state (not too many cases are beiag flagged; of the 
cases that aore flagged, most of them are indeed untimely cases). . If, <?h 
the other hand, more than 10% of the cases are flagged, there is a problem 
in the .service process itself^ since these cases ynave been judged as un- 
timely and there are too many untimely cases f« efficient moftitoring and 
efficient operations > This calls for an examination of the service 
process itself, perhaps ysing the..decision/suppor3: systott to analyze 
the-- state caseload prbcess and pinpoint tuielines.s issues in relationship 

rion, this problem jnay call foi;^ 
i^ard adjiistment of the times allowed in statuses, to flag fewer cases. 
However, checking for the appropriate times must be dond in another itera- 
tion so that a check can be made as to whether both the 20% and 10% , K 
st^aiSards are met for a given hew standards level. ^ ^ i . 

Finally* the state^iaould routinely flag between S% and 10% of its 
cas.es, to* assure that^laggin^^ standard5-i»e set low exiough. If less than 
S% of cases a» flagged, the^ standards should be made more stringent 
(allowed times in status decreased) before the next round of review. 

Using t'his approach, -states can adjust their times iii status standards 
tqjwards or downwards to be more meaningful and result in an efficient 
pro<;ess t|iat spots problematic cases without excessive monitoring . 
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The raod^i illustrates how information from this procedural . standard 
can be used w4th othet program information to refine and improve state 
ponxtoring systems. 
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STANDARD 12: IWRP ' ^ ' * ' . * 

VR 'shall OTOvide an Indivigualized tfrxtten Rehabilitation 
program for each applicable client, and VR and the client 
shall be accountable to each other for complying with this 
agreement . • 

Several aspects, of the Individualized Written Rehabilitation 
Program are addressed in this Procedtiral Standard : (a) compliance 
with the requirement that an ITOP be fully developed for clients accepted' 
for services. or extended evaluation; Qs) assurance of the protection of 
client rights and client awareness ol.the remedies available for mitigating 
dissatisfaction; (c) joint client/counselor development of the job goal and 
the service plan; .Cd) mutual client/counselor responsibility for follow- 
through on the agreement and annual review of its progress and appropriate- 
ness; and .(e) the appropriate handling of plan revisions. 

This standard beays a relation to the RSA goals of compliance and qual 
ity case services. Obviously, given the regulations mandating provision of 
an IWRP to all" accepted clients, this standard's relation to the compliance 
goal is clear. While the regulations concerning the imP stipulate compli- 
ance with the provisions of the law, elevating the issue to the level of a 
procedural standard will ensure compliance with the legislative in-^ent of 
the IIVRP. 

Inclusion of this standard could be justified^'simply on the basis of . 
tlie .Strong .regulation/ regarding compliance with the IMRP provisions of the 
I97S Rehabilitation Act. However, perhaps an even more important reason- 
to include this standard is the fact that research has shown a positive 
association between compliance with the HfllP requirements and successful 
outcomes o£ the VR process Since research has supported the premises 
underpinning the mXP by showing that the process and the possession of the 
VHKP affect client outcomes positively, adherence to the ilifRP requirements 
becomes a powerful norm "for quality case management in VR, as ^ well as a 
protection o^ client interests apd rights. The case review serves as 
the data source for this standard. 



• ^Berkeley, Planning Associates/ Implementing the VR Act of '1 973: The ' 
VR Program Response , p. 59. (1978) 



STAM)ARD 13: GOAL PLANNING 

Counselors shall make an ejSfort to set realistic 
goals for clients^ Comprehensive consideration niust 
be given to all factors in developing appropriate 
vocational goals- such that there> is a maximuni.vof~ ^ 
correspondence between goals ^d outaomes ; comp'et^^ 
tive goals should have competitive qutcomes and non* 
competitive goals should haVe noncompetitive outcomes , 

Competitxve-eaployment may hot be, the appropriate placement for all , 
clients.. Nevertheless, VR regulations require that all placements be into 
"gainful activity^ and that placements be consistent with the 'clients' \ 
». "capacities and abilities," whether, in competitive, sheltered, or noncom- 
petitive employment. • ^ 

There is much speculation in the field over the abxise of "homemaker"' 
and "unpaid family worker" categories, specifically regarding the use of 
these categories to erusun success rather than because the placement is 
appropriate. While maxiijiizing th^ propQrtion of sucx»ssful closures (as 
in data element 3Ci)) is xB^ortant to the purppse'of VR, it does not .ensiire 
that noncompetitive^ placements are suitable for the client. This standard 
addresses the cdhcem that noncoaqjetitive closure categories not be used 
to. salvage "successes;' for clients who wearei unsuccisssful in their planned 
competitive goals, • ^s0^''^' '"~ % ' ^ f - - ^ 

However, this standard is not intended to "fceere" counselors and 
their clients into goals as set out in the original IIVRP, y<SiL:h an effect ' ■ 
would be a misapplication of the IWRP proce«^gi^fl!riWRP «• intended to^ 
be a statement of a realistically attainable goal which, if necessary, 
can be modified for a variety of valid reasons ^ thtf ciieAt progresses* 
through the .VR process. That is^the IWRP serves as a guideline rather 
than as a hard and fast rule* , * v 

As such, state agencies should not use the results found for the ' 
Standard in such a way as to.'overenqftxasize the importance of matching th* 
outcome to the goal. This would serve 'as a disincentive to setting ^ 
ambitiojis (i^e., competitive employment) goals in the original BfRP, and 
would reduce the flexibility of the counselor in refining the goal in 
response to client progress ^during the rehabilitation process. Instead, if • 
•'problems" emerge on, the standai^i^ the results should be used in conjunction 
with data on client characteristics and services provided to investigate^ how 
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counselors can ^be more effective in the task of ''fitting" clients' potei^tials 
to feasible ultimate outcomes. In this way, the standard* is used apprp-i 
priately to facilitate effective goal-planning rather than, simply to fpais 
•m whether goals matched outcomes. 

Standard 13 uses four variations on a common theme as data elements: 





# of 26 closures with competitive goal 


(i) 


AND conroetitive outcome 


^ 9f 26 closures 




# of 26 closures with competitive goal 


Cii) 


BUT non-competitive outcome 


^ of 26 closures 




of 26 closures with non-competitive goal 


(iii) 


AND non-competitive outcome 


* of 26 closures ' 




' * of 26 closures with non-competitive goal 


Clv) 


BUT competitive outcome 


# of 26 closures 




* 
* 


* 
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III. RELATIONS AMONG THE PROGRAM STANDARDS 

The mmber o£ performance standards data elements (21) . creates cer* 
.tain problems in designing- and implementing the standards system. The^ prob- 
lems revolve arotmd which standards and elements ^to ec^hasize, realizing 
that some choice wi 11^ have to be made between improving on one standard^ 
ov«ir another, or improving a little on each. The problep is exacerbated 
when iiliprovement on one standard may be at the expense of another standard, 
i.e., with decline on another standard. 

The varxoix5 data elements represent diverse gpals, some in conflict. 
With a single objective function (single goal), VR may be expected to max- 
imize on the measure (e.g., close as many clients as '*26s" as possible). 
But with multiple objectives, as represented by the 'standards and, data 
elements, conflicting in many subtle and not-so-subtle ways, a system of 
attainment levels can only indicate desired achievement on. all element^ . ' 
Among- elements are trade-offs, however. For example, an agency might Jiax- 
imize its benefit-cost ratio by reducing its coverage rate and by creaming. 
As a result, success in achieving one program, goal could be counterproducti 
to success on other goals. 

The reason this problem arises, of course, is because the basic VR 
program has several conflicting^ although legitimate , performance and ser-^ 
vice objectives. (To paraphrase one regional official visited during our 
study, VR is really many programs, each with distinct goals.) 

The problem of conflicting mandates can be illustrated with a simple 
listing of hypothetical '^maxims'* for the VR program (keyed to the revised 
dati elements) : 

(1) The basic purpose of VR- is to assist disabled individuals in 
obtaining '^maximum participation in gainful employment, con- 
sistent with his or her abrlities.*^ (Standards 3 and 6) 

(2) Ideally, when the client finishes VR, he or she will be able 
to compete with nondisabled persons for jobs paying %z least., 

• entry- level wages* (Standards 4 and S) ^ ^ 

♦ — 



so 

(3) ' Not all clients will be able to achieve competitive employment. 

For such clients, VR may provide services aimed. at obtaining 
noncqmpexitive employment: homemaking, sheltered work, and 
other unpaid wcrk. However, VR shall make every re^^onable 
attempt to identify the possibilities for obtaining competitive 
enployraent before deciding on a noncompetitive job goal. Mso,&. 
in such cases clients shall have obtained some type of benefit 
fifom VR, whether vocationally (e.g,, enhanced job skills), or 
nonvocationally (e.g., enhanced abilities for self-care). 
(Standard 5, dat^ element iii; Standard 7, data element iii) 

(4) VR shall serve as many eligible clients as it can. (Standard 
1, data element i) 

(5) VR shall use its resources as efficiently as possible. (Stan;- 
daxd 2) / 

(6) VR shall give priority service to severely handicapped individuals 
(Standard 1, data element ii) 

Each of these maxiais . concerns the outputs of VR, and there are con- 
flicts even among these six. For example, assme that competitive employ- 
ment is indeed the ideal outcome. Since competitive employment requires 
greater skills than noncompetitive employment, then, in general, greater 
effort will be required of VR Cixt terms of time' and cost) to achieve com- 
petitive mployment outcomes. That is, costs go up (and '^efficiency" goes 
down) to the extent that competitive employment outcomes are emphasized. ' 
There may be"^ another conflict between serving the severely disabled and 
the goal to serve the most clients. 

.The discussion below presents some empirical findings* on the relations 
among measures of performancB to underline the difficulties that multiple 
measures c^use. Then, two approaches to the problems of multiple measures 
are presented with their shortcomings noted. Last, the implications of^ 
continuing with multiple measures for the standards system are laid out. 

REUTIONS AMONG MEASURES OF PERFORMANCE ^ _ _ 

A data base of all the standar4s data elements is not available for 
all agencies (since several new data collecS^on instruments are involved 
and since the standards^ pretest was carried out in only six state VR 

■ ■ ■ • . 58 



SI * 



agencies). However, the dita- is available for some «bf the daifa elements, 
allowing analysis of some of the relationships among VR performance measures. 

In JCnuce, Miller, and Cope, relationships among several measiires of 
inputs, process, and outputs were investigated for the S4 states and U,S, 
territories. Several of the measures used are found in the standards 
data elements, and others are very similar. The bivariate correlations . 
anong some of these measures for 1968 an4 1969 are shown in Table 3 . 
The authors state: 

Both the high levels. of rehabilitation rate and rehabilitant 
salaa^ are desirable program outcomes- Yet, the resxilts 
suggest that these outcomes may be incompatible with each * , 
other. Such an incompatibility is highlighted by the opposite 
relationship that the two output variables ithe rehabili.tajtion 
rate and rehabilitant *s salary) have to the number of cases 
swved. Where more clients are served, the rehabilitation 
rate is higher but rehabilitant salary is lower, ConYersely> 
when fewer clients are served a hi^er placement level Creha- 
bilitant^s salary) is achieved. Parenthetically, it is noted 
that volume, as measured by number of cases per 100,000, is 
associated with lower rehalDilitation cost. 

The inverse relationship between rehabilitation rate-^d 
rehabilitant's salary has special implicaticns for program 
evaluarion* Many pro g rams that look good on one of these 
outcome variables will look bad on the other one* This 
finding does not nftcessarily imply that programs high or low 
on either of these variables are good or bad^ However,, the 
results "do strongly support a position that the two- kinds 
of programs have different resources and strategies . Those 
agencies with high rates tend to have more financial resources, 
work with more clients, rely more on workshops, and keep 
clients in the caseload for a shorter period of time. Those 
with lower rates tend to deal with fewer cliaits, be more 
selective in acceptjing clients^, keep them in the program for 
a longer period of time, and provide them more training. ^ 
Therefore, examination and evaluation of a program on the 
basis of only one criterion could lead to erroneous con- 
clusions about program effectiveness.' (page 137) 

In Dodson (1978) , factor analysis was u^ed to investigate the 
rljlationships among measures of outputs, and among measures of perfor- 
mance for the states for 1970. The*results of the factor analysis for^ 
the performance measures are shown In Table 4. \ , 

A factor analysis of all. eight performance measures yields three 
factors. The first ha^ high loadings on.% with earnings *at closure C-93), 
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Table 3 



Correlation Among Measures -- iCnuce. Miller, and Cope 



^Measure- 



1 — — 



1. Clients served 
per 100^,000 pop- 
ulation 

2. Expenditure per 
26 closure 

3. RehabrUtation 
rate pet 100,000 

^^popuia^on 

4. Average earnings 
at closure for 
26 



J 



^96^/196? 



.897.92 



-.39/-.S6 



-.34/-. 29 

.06/. 30 
-.39/-. 39 



/ 
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Table 4 

Factor Analysis of Outpjat Measures -« Dodson (1978) 



• 




Factor ' 


* 


Measure 


1 


'2 


. 3 


1* Percent 26 vrith earnings at . 
closxire 








2* Average earnings at closuijp 








3. Percent 26 with competitive ^ 
employment * . ' 


,86 




y 

•/ 


4 Percent homemakinir 


-.93 






5* Increase in earnings from / 
referral to dpsiire 




.81 




6. Reduction in public assis* 1 
tance 








7. Percent with public assis- ^ 
^ tance at closure 








8* Benefit cost ratio (crude) 






.?9 



r 
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competitive ^loyment C*86), and homemaking (-.93). The second factor 
has high loadings on earnings at closure (* 93) and on increase in earnings 
(.81), reflecting the high correlation (.92) between these two measures. 
The third ha^ a high loading on benefit cost ratio (.^9). These factors 
account for the following percentages of the shared variation; SO. 6%, 
31.1%, and IS. 3%, respectively. Overall, these three f^ictors account for 
71.1% of the total variation, ol the individual measures, only reduction 
in ptiblic assistance (4.5%) and H with public assistance at closure (23.1%) 
have- less, than SO* of their variation e^dplainid by these three factors.- 

thus, the percentagepof earnings or competitive employment (as in 
data element 4i or 5i) comprises a very different dimension than the 
absolute level of earnings (as in data element. 4ii) . The cost benefit 
ratio presents again 'another dimension*^ ' . ^ 

The main conclusions to be drawn from these analyses are: 
^ some of the standards data elements are positively related, 
so that an agency doing weU. in one data element will likely 
be doing well on other data elements; 

♦ some of the standards data elanents are unrelated, even within 
. the same standard, so that an "agency doing y^ell on ,one data 

eleoent will not be telated. to its doing well on other data 
elements; ^ 

• some of the standards data elements are negatively related, so 
that an agency doing well on one data element will likely do' 

less well on another data element; and 
r the. attainment of VR agencies with regard to the Performance 
and Procedural Program Standards is clearly multidimensional. 

APPROACHING THE PROBLEMS OF THE REUTIONS AMONG THE PROGRAM STANDARDS 

There are some possible approaches to the problems of the relations 
among program goals. Two are discussed below: the use of a- composite 
scal<| and th^ specification of a hierarchy of standards arid data elements. 
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Hrhese dimensions have a correlation of zero, since orthogonal rota- 
tion has been performed; when oblique rotation was tried, the correlations 
between dimensions stayed near zero. 

■ ■ ■ ^ , , . 



ss 



• Use of a Composite Score • • ^ 

The measurement of the 13 different standards^ will necessarily in- ' ' 
volve discrete measure^ of an agency's attainment on 21 data elements for 
performance standards plus procedural standarc|^. This has raised the 
cruestion of the desirability of the development of some composite measure 
that would enable one to quickly sximmarize the status^ of any given agency's- 
attainment or to easily compare across agencies. 

^ There are })oth technical and conceptual issues tor^be addressed in- 
developing or considering the merit of composite measures. Technical 
issues J once the decision to use a composite has been made, are two: 
Standardizing different xmits of measure Cso that, for example, percent 
.competitively employed, dollars per successful client, ratio of closure 
to referral earnings can be. combined in one measure) , and weighting of 
the indivic&ial elements comprising the composite measure (is the propor- 
tion of successful clients of equal weight to the cost -effectiveness of 
serving clients; are these of equal weight to an ove^ll program benefit- 
cost ratio?r* At least cfhe approach to the stanc^rdization of scores has 
already been applied to VR standards Ci*e.,. the current nine standards, 
not these^proposed revised standards) . That approach was outlined by Perry 
Levinson, the Research and Evaluation Specialist for Region IV, in his 
adaptation of the Profile Analysis Technique to the development of Per- 
formance Level Scores, using stanine (standard nine) scores for each . 
measure t Given the desire to produce a composite performance score, that 
approach or some other variant could be used. However, tt^ie units of 
staiidardized scores are^difficult tti interpret and thus c6ald result in 
some confusion in the reporting system. The second technical issue, however, 
that of weighting, is more complex. In fact, it is not'solely a technical 
•issue, but a very serious ^licy issue. Wxo is to be Responsible for 
establishing which standards, and which measures within each standards 
are most is^ortant? How are the specific numeric weights to be assigned? 
BPA believes that this would represent, for the VR Program Standards, a 
very difficult task. 

There ar^ serious conceptual issues in' addition to .the technical 
issues outlined above. Each data element has been selected, for its 
ability t^/n^^asure a discrete aspect of that standard. Thus, it is 



' important^ to T)e able tO/^bse^e a given agency^i attaS^hinent both* in. terms 
of its ability 'to produce competitive^ Anpioymei^* closure and in terms of 
its clients' ability, to. retain ;benft£its for an extended period after ter- 

' ilnation of services, .Sijnilfcly,. it is 'important to know the cost-bertefit 
^and cost-ef£ectiven^s'$.with^ which the agency can adai eve these objectives. 
The use of a composite measure', while it appears to facilitate comparison 
across- agencies', actually masls^s- or 4pses information. Two agencies, each 

. with a 'qomposijce score of 60 on. a scale of 100, -may have very different 
strengjffis and .weaknesses: onp fiiay have very 'high client satisfaction, 
but very high costs ,also; th# other may have. very high cost-effectiveness, 
but veijy^ low satisfaction of its' cli^ents. An example^-of this problem is 
shown in Table S : ,The levels of attainment for two data elements, both 
expressed as percentages, are given for six hypothetical agencies. If 
equal weights of '1.0 are used for each data el^ent, the first three • 
agencies show 'the same composite scores t60%)* and the next three show 
40%, As the table shows, the' composite score masks attainment on the two* 
data elements., which is very different.' The overall attainment of agency 
D IS clearly less than that 6i agency A. However*, despite the lowi»r com- 
p^Dsite scores, .is it clear that agencies E and F axe wcjrse than agency A? 
The primary purpose of .the system that BPA, RSA, aad CSAVR have taken 
^^zxns to develop is to examine differences across agency attainment , 
reasons for problematic attainment, ^ and thus to be ajjle to provide 

* guidance -ecj how to improve attainment . The con5)Utation of composite 

. attainment measures does not contribute ta this end and, in fact, is 
likely to serve to divert attention from this pxirpoie, which involves 
careful interpretation and thorough .analysis, to the facile purpose of . 
?*ranking" and "rating" ag*icies. ^ - * . > 

We do believe, however, [that underlying the quest for composite 
scores are some legitimate concerns. ^The concerns are that the individual 
. standards cannot be lo.oked at solely in isolation, that there are relation]^ 
' ships among standards, that there are trade-offs in the ability to show 
successful attainment on the standards, and that a "systems" or' "holistic" 
approach to .considering attainment' on the standards is needed. Thus, an 
agency that does well in producing competitive, high-wage closures may hot ^ 
do so well in the proportion of all clients that are "successful" — quality 
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■ Table S 
Pfobl ems with a Composite Score; 
Hypothetical Attainment on Two Data Elements 



VR Agency 



Data Element 1 



Date Element 2 



Equally Weighted 
Composite Scdre 



A 

B . 

C 

D 

E 

F 



60% 
95% 
40% 
40% 
70% 
50% 



60% 
25% 
30% 
40% 
10%- 
75% 



60% 
60% 
60% 
40% 
40% 
40% 



\ 
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may involve a trade-off in quantity. aMuraeroiis such trade-offs exist in 
terms of the standards. Attention to this issue is critical. The point, 
however, is that use of composite measures does not elucidate th^ trade-offs 
and relationships, but rather hides thorn . The high -quantity- low-quality 
agency and the high-quality- low-quantity agency ma/ well have the same com- 
posite score. ^ 

The appropriate format for looking in a "systems" manner at an agency's 
attainment is the decision support structure , explained in Chapter V and 
^ illustrated in detail in Appendix A of this report, that wo have devised ^ - 
as the framework for the standards. Rathe^ than subsuming the standards 
under each other, this system allows the analyst to statistically "control" 
for one standard in assessing attainment on another. This logical structure 
is far more- comprehensive, in profviding the ability to control for elements 
of VR outside of the standards. /Thus, in understand?jxg performance. on 
measxrres of quality outcome for l^andard 4 (e.g., wages), we can look not 
only at an agency's performandte ontSe quantity of successful placement 
(Standard Z) , but also at client mix (severity of disability), economic 
condition of the state and region, and othe-r important factors not part 
of the standards, at all. Techniques such as the Profile Analysis Tech- 
nique and more complex multivariate (such as regression) analysis tech- 
niques also allow examination of multiple measures and permit RSA, state 
program managers, and outside evaluators to see relationships among stan- 
dards, determine what is facilitatGig or hindering good attainment, and 
identify the type of program changes 'that will lead to improve attainment 
— th^s is exactly what the composite score will not do, 

V 

Hierarchy of Standards and Data Elements 

Another approach less demanding than the composite score is the 

specification .of a hierarchy of standards and data elements. In spite 

of the many criticisms of the composite score just discussed, the most 

tei^g problem is the near impossibility in getting weights developed 

and agreed on. A less demanding approach would be to bypass exact weights %. 

and instead identify a hierarchy of sta^idards and data elements. 

r * • 

A very simple form of ^ this hierarchy wctild be for the state VR agency 

or RSA to raadk the^data elemei^ts, ^wi;th the highest ranking data element 
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at the top o£ the hierarchy, and so on. With this hierarchy, state VR 
agencies coiold first focus all their efforts on the top priority data 
element, then on the second, through the last. 

Even simpler would be for the state VR agency or RSA to identify one 
high priority data elementt with all the other data elements secondary* 
However, in either form, either when ranked closely or in the secondary 
set of data elements, some way would have to be found to express the trade 
off between zero or negatively correlated data elements* 

Hpw realistic is it to develop such a hierarchy, and to obtain a 
consensus from' all involved parties,^ including the Depaartment of Educa- 
tion, the OIB, the Congi^ess, the states, CSAVR,,, consumer groups, etc.? 
One way to'lDegin to develop a hierarchy would be to recognize that dif- 
ferent data elements may be seen to serve different functions: 

# The most important data elraents are those that serve as the 
key indicators of success on the program's priority objectives. 

^ Other data elraents serve as indicators of aspects of perfor- 
mance o£ interest to RSA, rather than of priority objectives. ' 
The reasons that a data element may be of tangential interest 
include: the output represented by the data element may be 
beyond the control of VR; or, vre do not yet understand the 
causes of success on the data element, yet we are still in- • 
terested in observing states' performances; or, the methodology 
has yet to be developed to a level suitable for use in assess- 
ing state performance. 

• Other data elements or their numerators or denominators are 
intended less as management tools than as ways of presenting 
VR to the outside world: their value lies in the areas of 
public relations and lobbying rather than ^as indicators for 
management (client satisfaction coulcl be seen in this way) . 

The implications of contintiing with the multiple measures without 
composite scores or without a hierarchy of standards are very clear- 
The effect is to weight each data element for each standard equally. 
Also, the state VR agencies are given one explicit direction for trade-^ 
offs between data elements. However , in objectives set by each agency, 
implicit weights and implicit trade-offs will be reflected. 
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Chapter IV reviews the issues involved in setting agency objectives 
on the standards; the levels of attainment set for each data item thus 
become the giiideldnes for directing agency performance. Chapter V dis- , 
cusses the diagnostic power available by combining different measures in 
the decision support system; such analysis, should be undertaken if agency 
performance shows problematic achievement on one or more of the dat^ 
elements. ^ \ ^ 

BPA suggests that, two steps be-t^en to acidress the question of 
weights. First, in the decision support system, the relations be'tw^n 
the attainment by agencies on the data elements should be examined empiric- 
ally each year. In this way, the continuing existence of trade-^offs can 
be verified. If attainment on. all of the data elements .shows very high 
positive intercorrelations, then the problem is resolved. Zero or negative 
correlations will indicate that the problem still exists.. For example, we 
might find that 100% "impact" can be obtained only if we. are willing to 
settle for* 60* coverage (i.e., by sacrificing 40% coverage) and 40% effiq- 
iency. Infinite other variations would exist ♦ This information^ on the 
range of possible trade-offs could be presented to policy makers, and 
throughr some process of^ consensus building the acceptable trade-offs could 
be determined. The ma-iti advantage of this approach is that specific pro- 
gram choices would be identified. 

Second, as the standards system operates ove* time, a .state VR agency 
or RSA will be able to examine the levels of attainment reached -overall. 
As these levels increase perhaps disproportionately, certain data- elements 
could be targeted for enqjhasis. 

Together, these two approaches afford a practical rolei for st^dards 
in the operation of state programs.* The measutes and their relationships 
can be refined ^ased on knowledge gained through use and experience. 
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IV. SETTING VR AGEN CY OBJECTIVES VIS-A-VIS THE STANDARDS 

^ 7 ' 

, Once standards and associated data elements measuring the goals 'and 
'fimctions of the VR system have been specified, some way must be devised 
to set levels of each data element as the objective of -each VR agency. 
Today, the nftttsuris for performance come retroactively, from comparison 
with other states' outcomes • This new system calls for prospective goal 
setting paying that a VR aigency should make sure, for example that rehab 
iiitated clients shall evidence economic independence is fine. But, for 
the data element ''percent ?6 dlosures with weekly earnings at or above 
federal Hnr-^Tmnw wage," no level Us obvious. SJiould the agency have 100% 
of its clients at or above mi.niTmT7n wage? 80%? 60*? Some way must be 
found to set the level that a particular VR agency should strive for. 
*^ The discussion of setting VR agency objectives proceeds^ as follows. 
First, the question of whether there should ^be objectives is raised • 
Second, three existing methods for settifig objectives are described. 
Third, the- dimensions of a method for setting objectives are outlined* 
Lastly^ a new proposal for setting objectives is put forth^ 

" SHOULD \hERE be OBJECTIVES? " 

• ^ _ ^ 

A standards system could operate without objectives. However, would 
such a standards, system result in higher levels of aiTtainment on the data 
• • elements? Without objectives, there would be no firm guides to state VR 
agency decisionmakers. They would have measures from previous years, and 
a vague mission of cioing better (or, in these times, maintaining perfor- 
mance) . Moreover, the state VR agency would not have any signals as to 
whether the decisions and actions of the previous* years had .the desired 
effects! No infoxmation on the level of effort or amount of change would 
be available because there would be no yardstick to meastcre progress by. 
Without an objective, very simply, progress is hard to measure. 
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EXISTING APPROACHES TO SETTING OBJECTIVES ^ 

There are some existing approaches to setting objectives. They are 
reviewed below to identify the quiropt^tat^-of-th^-art. They include: 

• the method used in 'the ctirrent standards; 
4 the method used in the SSDI/SSI-VR Special Program; and 

• the method previously proposed by BPA for the revised 
Standards* 

Approach of the Current Standards: National Averages 

National averages are used in setting objectives in the current stan- 
dards. There, statistical distributions of particular data elements ar.e 
developed each year, and those states that fall outside the acceptable 
bounds are identified* Several alternative ways for identifying cut-off 
. points are used, including a central t9ndency based approach Csuch as - 
one standard deviation from the mean) or percentaging (such as. bottom 10%) 
or ranking Cbottoo 10 agencies) • 

On the positive side, this approach provides a relatively easy way to 
identify those states with below average performance. Objectives Isased on 
nationi;l averages allow the targeting of states that may need technical 
assistance or at least yrurther evaluation. Additionally, since the use of, 
national averages provides an easy way to view aggregate national perform- 
ance, this approach provides one way to identify overall program pla nnin g 
needs. 

There are many negatives associated with thxis approach. Primary among 
them is the establishment of a model of perfomance based on a statistical 
average Cthe status quo), which may or may not represent an appropriate 
target for agency attainment • ThjS approach, by providing no a priori 
targets for states to shoot for, is not likely to motivate states^ to^ change 
present practice. It inherently places some agencies outside the accept- 
able performance levels because of the statistical' construction of the level, 
even if no ifelicy or management rationale distinguishes their |>erformance 
as unacceptable. Beyond this, the approach may recpure xontrols to ^j^t- ' 
for exogenous factors that would affect., performance, since a national' 
comparison may not be appropriate for particular agencies, on particular 
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data elements. Even with improved measures of central tendency, these 
problems still hold. ^ 

« 

ApoToach Used in the SSDI/SSI^VR Special Programs: Norms 

For the SSDI/SSI-VR Special Program, RSA took the tack of directly 
specifying the level each agency should reach on a number of data elements. 
In a December 21, 1977 Information Memorandum (RSA-PI-78-10) , four cz$e 
closure, performance data. elements were considered: percentage SGA (signi- 
ficant gainful activity); percentage cases meeting SSC (special selection 
criteria), receiving .significant services, and achieving SGA; cost per SGA; 
and cost per 26. For the percentage SGA, an objective of 80% wSs ^et. The 
basis for this level is not discussed, but seems to be what RSA thinks the 
agencies should attain. , _ _ 

Approach Proposed for the Revised Standards: Progress Levels ' 

The appToa"ch recommended by BPA in 1978 is a progress approach, which 
utilizes a retrospective performance classification method in order to 
establish a baseline assessment of a given state's attainment on any given 
data element. This baseline assessment 'categorizes individual state VR 
Agencies into high, medium, or low levels of attainment for a specific 
year. "High" is defined as performance equal to or above the national' 
average performance for. any data element (or, the national mean for any 
data element) . Medium performance is defined as performance between the 
national .average for any one of the data, elements and the first standard 
deviation below the average, and low performance is that performance 
beneath .the &x^t~ standard deviation below ,iJie national average. This 

' categorization' is then the basis for prospectively establishing expected 
attainment for the coming year, .based on th'e history of attainment of the 
agency and fethers on a particular data element. . ' 

A state would be expected to perform a specified fraction, or per- 
centage, of the national rate of performance change on each specific data 
element. The fraction or percentage will depend on which performance 
claas 9. state fal^s into. The national rate of performance change is ' 

■nothing more than the average percentage^ rate that all states (combined) 
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have improved or declined between specific years on each data element. 
For positive national rates of change, states classified in the high 
c^egory' would be expected to increase performance 75% of the national . 
rate of performance change on any givenrd^a el«nenrt ^hile xhose ^>t;atesr 
classified in the medium performance categories would be expected to im- 

« 

prove 125% of the national rate of performance change. Those states 
landing in the low performance margins are expected to improve 150% of the 
national rate of performance change. For negative national rates of 
change, the percentages shotUd be reversed Cthe worst states can not 
afford to slip as much as the "high'* states) • The rationale behind this 
is that such expectations require states doing well to maintain their 
level of performance while not pushing them to unreasonable goals- How- 
ever, those states in the medium and low categories are expected to in- 
crease their performance relative to their national standing and the 
national average. This provides direction to each state regarding what 
is expected of it relative to all states and subsequently provides an 
optimistic direction fo/ the national rehabilitation program. 

A mi PROPOSAL FOR SETTING VR AGENCY .OBJECTIVES 



During t|ie pretest, BPA reviewed a. number of other alternatives to 
setting levels. BPA proposes a new approach for setting VR agency objec- 
tives, an approach that shares some of the aspects of the L978 proposal, ^ 
but diverges from that proposal in other Jfays • 

Since si prospective method Jor Setting VR agency objectives has not 
been in effect, '^the VR system has no experience to draw on in evaluating 
the various possible methods* We suggest a workable approach in the short 
run, with monitoring, e^luation, and revision of the approach after the 
system has been operating for a time* ^ 

As part of the initial objective setting i^e levels of attainment 
on each of the standard data elements for the previous year, the objec- 
tives for the year, a^ historical performance of the a|ency will be . 
reviewed. Also reviewed would be the values of all the component? of the 
data elements and assf^ciated information data elements* The levels of 
attainment, negotiate^ objectives, and informational data elements could 
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also be reviewed for comparable agencies, exeinpJary agencies, and the 
nation as a whole. All of this information could be«;vailable in the 
reporting system described in Chapter VI. 

A negotiation process could balance what the state' VR agency thinks 
it can attain versus what RSA- would like it to attain. Under such an r 
•approach,. the objective is unlikely^ to be viewed as infeasible or unreal- 
istic by the state VR agency. 

Once the new standards system has been operating for two years, the 
method of setting' objectives' should be evaluated. One part of the^alu- 
ation wSuld be a comparison of the objectives set by the various technical 
methods to the negotiated objectives, to see if these methods would have 
made a difference." I The major focus of the evaluation should be on whether 
'the levels of attainment on the data elements have increased. Jn addition, 
RSA should canvas the state VR agencies and Regional' Offioe-al^f on how 
the objective setting process is working. At that point, another method 
might be chosen, or the^ proposed method continued. 
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V. INVESTIGATING PROBLEMATIC ATTAINMENT : • 
. DATA-BASED -DECISION SUPPORT SYSTEM e ' ' ^ ^ 

Y state's performance on the data elements should be compaxed to the 7 
performance levels set for that period. Some agencies will h9t-;have met^- 
some o^-their objectives set for level of attainment on the^^stai^ards • 
The system does not stop with this comparison or gr.ading but ^tead moves 
to investigate the problematic attainment and to identify corrective actions 
as part of the decision support system . The puaT>ose of the decision sup- 
port system is to cldse the gap between reporting on the standards and ^ 
actions based on the standards. The system should: 

provide an ability to pinpoint causes for problems ii\ attainment; 

identilfy strltejjies leading to enhanced attainmejit; and 
' identify appropriate policy recommendations, and pwgram actions 

that c^n be taken by state agencies, RSA, or Congress, based on^- • 

the analysis a^ aimed at improvement in agency attainment • - ^ 

Achievement of these objectives requires synthesis of first-hand 
familiarity with pr ogr am operations, analytic techniqxies, and a sensitivity 
tfr policy concerns. Sensitivity to pjgjLicy concern^ is a most important 
consideration in terms of ' the overall design of the sxibportive^. evaluation 
system. Decisions are made by program managers, be they withi^i RSA^oii 
within state agencies. The standards valuation system is designed to 
inform decisions aimed at alleviating observed prbjjlems in agency. attain- 
ment. As such, the tfverall sxqjportive evaluation 'system must^ first and 
foremost address the information needs of program, managers* What this^ 
means in practidti^ terms is that, at a minimum, program managers niust be ^ 
provided with informit ion that is: , • * >^ 

#' releyilit to ,the issxies' Cfe;^. , problems) under consideration r 

e ' qljdckXh^^i easily intd^ ^ . - 

timely; and' . , / 

♦ suggestive either of an immediate policy response to the pr:oblem, 
or of furtheif^investigatzio*r:needed' b appropriate^, tespoiise ^ 

can be formulated. / . - v * . 



The' basic flow of the decision sxipport system is shovm in Figure 3. 
Problematic attainment, where 'an agency is unable to meet its objective 
for a particular standard data element is the, signa^l for the pjrticiess to 
start. -Problematic attainment is, flagged in the reports. and the 

state VR agencies initiate investigation of the problematic attainment • 
I^they are able to identify problems and possiblje corrective actions, 
, then implementation is the pext step. If not, then njore ^extensive investi- 
gation is called for* Implementation of the corrective actions will a£fect 
state VR agency operations in the^ next cyc^ of the standards system. ^ . ' 
a' result of the corrective actions the agency may be^able jio meet its,-ob-, 
jective?^ Otherwise, the cycle starts anew. • \ 

As noted, the investigation of problematic * attainment has been broken 
into two parts: . • ' 

# basic problfflg iden:tification 'carried out by VR and R§A staff 

. ' and the state VR agencies, using the standards reporting system 
plus agelicy taiowledge of program operatiofns; an^ ^ 

# evaluation research , carried xiut by ,evjiluation staff or byj^5r^ \- 

side consultants, using agency dalia and-\btHer data bases or 

gn-sita in:^stigations and case reviewi^r.^as need^^ 

This d^^ter is organized as follows: \' \» ^ 

0 a general model of the thinking process that managers Would . 
' undertake to investigate the causes of pbroblematic attainment 

. ^^.^is presented first; ^ , - ^yr. 

^ two -exaisples of how the thinking process would be applied 

^ • to data on the revised standards are given next; 

# the two major levels of statistical evaluation research, are 
sumiparizei^^ " ^ ^ 

THE PR0CEg3 OP PRdBLBi IDBglFICATION " _^ . 

The ' process o^problem identification outlined below may be carried out 
within individual state VR agencies or by RSA. The information for.;, the prob^em 
identification will /come firom the standards reporting system as well as from 
the agency managers' knowledge of program operations ♦ The pi^cess consists 
of narrqwing in on problems by examining a selected set of data indicators 
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Figure^ *" 
The Flow of the Decisibn«Suoporf 'System" 
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when a problem is flaggeil* These '"second- level indicatbrs" will then lead 
to particular diagnoses of agency problems or Indicate further areas 'for 
investigation, once the problem is identified to the manager's satisfaction* 
At tha^ point, corrective action is formulated- Or, at any point in tracing 
out these problems further analysis in the form of eyaluatxon research may 
be reqjiired. This process is, lik^i-that normally illustrated by a decision 
tree* . Of course, the profess* of problem identification may lead dowi several 
paths at once. The point is to, do the analytical thinking and, utilise exist- 
ing information to identify possible problOTS and corrective actions.' Also, 
more than tvo paths may neecl investigation from a particular node, or more 
than three levels of indicators may have to be examined- Below, the speci- 
fics of this, process are further delineated* 

a^datat element shows problematic attainment, the first level^of 
analysis is to examine the components of the element, dissecting the ratio 
or measure into its separate parts, to pinpoint the areas, needing attention. 
For example, if the numerical value of a ratio is too largd> the problem 
may Jbe in an area of agency operations reflected in tHe nmerator (too 
large) , the denominator (too small] , or both. Conqjarison of attainment ' 
on the data elements or their components with that of other agencies with 
similar programs, -or historically, or on other^data items, can help deter- , 
mine the extent to which the indicator shows a real problem or if there is 
a good explanation foj the attainment. The goal in this analysis is to' 
seek explanation , or' the identification of which^ components or related 
meastires pinpoint the areas to be explored further. This analytical 
process may take several iterations before a cause is pinpointed. The 
first levels of the process are not to be seen as complex statistical 
analysis problems, but rather straightforward, simple program comparisons 
that allow people to progress through a decision-tree, diagnosing problems 
and using program information to reach conclusions about probable causes. 
Some branches of a decision-trerf process may l.ead to problems or investi- 
gations that require compllx- statisjical analyses, but only several leyels 
into the process. For exaiaple, using- a case of poor performance on an 
agency's "expenditures per 26 rehabilitation," an examination of cost/ ^ 
closure and cost/ case may reveal that the agency is. achieving too low a 
proportion of 26 closures or might suggest that the system is either > serving 
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clients too tlowly or not serving enough clients. Each of these possibili-^ 
ties- can al^ be explored at the sfcond level. These alternatives are shown 
schematically in Figure 4. "Third level" indicators may be needed to explore 
the selected alternative further, before deciding on a specific course of 
action. .The information necessary to address these issues may be found in ^ 
other data elements within the ''standards system or may suggest^the need for • 
more formal evalug^tion research. 

Table 6 , shows in more detail the decisipn-stepS in this ex^ie explor 
ation. The column headed "first level indicators" shows four possible 
combinations of two other indicators, cost/closure and cost/case, which we • 
suggest using in conjunction vjth an unacceptable Oiigh) value of data- ' 
elanent Ziii) . Depending on acceptable or unacceptable levels of these 
indicators, a different "scenario," or type of problem, is ,idefttified. For 
instance, if both of these indicators are "acceptable," then this' indicates 
that the agency is achieving a proportion of 26 closure^ which is too low. 
This can be confirmed by referring to data element 3Ci). If cost/ciosure 
is unacceptable, but the cost/ case is acceptable, then the agency is. achiev- 
ing too 'few closures. As can be seen here, this first level diagnosis 
leads to in-depth investigation of different parts of'the systan. The table 
shows the: types of second- and third- level questions, that could be^ pursued, 
depending on the initial comparisons ' and explanation. Appendix A -of this 
report contains tables like this for the other" data elements , plus step-ljy- 
•step di^ays to guide the manager or analyst through eack/ 'decision tree," 

At each level of the investigation, the goal should be to quickly and 
more finely hone in on the precise nature Ci-e-. cause) ,^f the problem. 
Depending on the findings generated by a given level of the analysis , the" 
program manager may* decide either: that further investigation is warranted 
before formulating a policy response; that the findings are adequate to 
suggest an appropriate respoijise; or that, despite the adequacy of the find- 
ings, no useful policy response can be offered (.e.g., due to prior institu- 
tionalj legislative, or funding 'constraints) . • 

The indicators used in the investigation of problematic attainment are 
grouped and sequenced in such a way as to answer increasingly detailed 
questions . ^ This allows '•managers and evaluators to go a fair distance in 
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Figure 4^ ' ^ 
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determining the natuje of the problem before needing recourse to more soph- 
isticated and tiae-consxjming "causal" analyses. This is not tQ say that 
other sophisticated analyses are undesirable or unnecessary. On the con- 
trary, they as often as not may prove useful to managers in pinpointing 
precise causes of problem performance. However, the advantage of this » 
model is that it allows the agency to quickly investigate and discard 
certain hypotheses' regarding the problem's cause, and therefore to more - 
quickly direct the investigation toward what seems to be the likely 
cause. Once the likely cause is identified through use of the indicators, 
the manager can direct the evaluation/research staff to conduct the needed 
causal- analyses. • * 

In the discussion which follows, we illustrate the chronology of 
thinking and analysis which should occur prior to the formulation of a policy 
response, using numerical examples from the MEU data collected for the pre- 
test or available from reports. 

EXAMPLES 

The discussion presents two working examples of the supportive evalua- 
tion process. These two examples include investigating an agency-level 
data element 2Cii), cost per 26 closure, which was used alsa for Table 6 
and a client-level data element S(i), percent 26 closures competitively 
employed. These analyses will include indicators from state data, the 
Standards data elements, and information contained on the R-30G, RSA-2, 
and RSA-101. The reader should refer to Table 6, and its branches, in 
reading through the first example. 

One. important point must be made before this presentation of the 
investigation process can begin. In out examples, we use standards pre- 
test data from the six Model Evaluation Units (MEUs) . These analyses were 
subsequently reviewed with the agencies' evaluation staffs to verify inter- 
pretations. The purpose of the examples is to offer a more concrete demon- , 
stration of the use of the system. The analyses we- present reflect th^ 
amount of information available in the pretest and should be interpreted 
only to the extent that it follows from the results presented. Moreover, 
there was only one year of data available in the .pretest, and there were 
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na^a priori data element objectives set by these agencies. Since the 
attainment of an agency on a data element should %e , interpreted by a 
comparison to agency operating goals, alow value on a data element might 
or might not indicate problematic attainment. With this in mind, let us 
turn to a specific example using data element 2(ii), expenditure per 26 
closure. 

EXAMPLE ONE: AN AGENCY-LEVEL DATA ELEMENT.' EXPENDITURES PER REHAB-ILITATION 
The Pr oblem 

— „ • . . 

Tliedata element in the example includes both cost data (the numerator 
is total agency e:q3e'nditure in the year) and impact data (the denominator 
is the number of 26 closures or rehabilitations achieved in . the same period) 
For this example, we will assume that an agency, which we will designate 
• as Agency A, has recorded an unusually high valuo for expenditures per re- 
habilitation. In investigating the performance of Ageilcy A on expenditure 
-per 26 closure, shown in Table 7, Ag^ency A's program, manager .looks at two 
other agency's information to verify that performance on this data element 
is indeed high: 

Table 7 > 
. Expenditure -Pgr Rehabilitati<^ ^ 



Agency 


-Expenditures/Rehalji litation 


A 


S 4,461.30" / 


B 


$ 2,408.64 


C 


$ 3,892.78 ■• 



From this table", we see that Agency A has a high cost per 26 closure,' 
compared to Agencies B and C, located in the same RSA region. In an actual 
case. Agency A -would be using additional infoxmatioh such' as -comparisons - 
-to past agency performance, to other similar agencies, or to other baseline 
data. ;n our analysis, we rely sol.ely on the pretest data and >^eref ore 
base our example oit comparisons with' two other states. x 

\ ' • 



First Level Indicators (see Table 6, /'First Level Indicators*') 

In order to analyza attainment on this data element, the program . 
manager must attempt first to determine which of the data elepent's 
corappneni:s explains "the problem^*: specifically, has the agency failed 
to obtain a large enough number of rehabilitations (i.e., the denominator 
is small relative to the numerator)?" Or, alternatively, does the agency 
have a problem controlling its various costs (numerator large relative 
to denominator)? It may appear that, in the coni^:it of this data 
element, the two issues are interrelated and inseparable: if costs p^r 
26, closure are high,, then by definition the agency has both "lost control" 
of its costs relative to the number of 26 's produced, and it has failed to 
produce an appropriate number of 26 's given its ejqjenditure level • Hot^ever, 
despite ti\e intuitive sense of this linkage, the numbers and cost ques- 
tions can and need to Be^ separated in analyzing, the datST element. This 

will be shown below. , . 

The method developed for analyzing this data element is a^ follows • 

Firs^, we make a preliminary assimction that the problem lies in the 

agency's " 26 factor^ '; that isT in the speed and frequency with w hich the 
agency produces 26 closures, only if the first aisaf second level indica-t 
tors sufgest no production (26 's) ijroblem, will the manager undertake a 
cost analysis. 

Thus, the first questitJn the manager asks is "why do we have so few - 
26 closures?" One way to analyze this question is to place it in^he con- 
text of a client flow problem. That is, the agency's low number of 26 I 
closures may be the result of some bottleneck or failure in the service i 
jnrocess. This is the approach taken below ^ which identifies four separate 
flow problems, (moving from the latest to. the eai^iest phases of the service 
process) : ^ ' ^ ^ • 

1. The agency is rehabilitating -too small a proportion, of 
its accepted closures. ^ 

2. The agency -is achieving too^feiv closures, in general, due 
tb bottlenecks in the service process for accepted clients 
('timeliness problem). Stited from a different focus,' the 

'accepted clients of the ageaQy arre spending Relatively 
, longer periods of time in the vaifious service statuses. 



77 



This reduces the pool of- clients nearing clostirei and thus 
reduces thfe pool of potential 26 closures, 
J. The agency is accepting too few clients, thus cutting off. 

the flow of potential 26 closures (intake problem) • . 
4* The agency has* both an intake and a timeliness problem* . 

The manager can test each of these hypotheses by looking simultaneotisly 
at two related indicators: total expendittire per closure; and total expend- 
ittire per accepted case. These two indicators use the same ntmeratprsj 
^total expenditures) as is used in the data element/ 2ii,: expenditure per 
rehabilitation. However, they ^'spread^^ the expenditures over larger groups 
and by analyzing the size of those larger* groups in relation to expendi- 
tures, the manager gets an idea of what (if any) kind of flow problem exists. 

The resxxlts of this investigation follow: / 

■ ■ '■ ^' - 

1 Tabie 8 

Expendimre Per Closure and Expeiiiditure Per Case 



Agency 


i 

Expenditnre/Clostrre 


Exoenditure/Case 


A 


53,428.60 


$1,573.69 


? 


1,757.51 


714.58 


C 


2,429.91 


1,108^ 



(From the RSA-2 and RSA-1) ; 

Based on Agency A's comparison to the other two agencies, the program 
manager concludes that this agency has both an unacceptably high expenditure 
per closure and expenditure per case*. The former suggests that clients are 
moving through the service statuses at a slow pace (a servd^ce bottleneck, 
due to a timeliness problem or to an* increased provisioit' of long term ser- 
vices). The latter suggests that the exi:jtence of an intake bottleneck, 
because a smaller caseload size win result ia a high average expenditure ^ 
per caie. • ' 



Second Level Indicator for Investigating High Expenditures Per Closure 

As a rough t^e'sr of the exi^t^ce of service bottlenecks (as evidenced 
by a high cost/closur*) , .the program manager tuns to the second level ^ 



indicator,, post-acceptance closure rate._ Thi's incicator compares the 
number of cases cloded durxftg the reporting period to the nuulber' of open 
cases. If this percentage declined over time' it vouid indicate a slowing 
trenk in the flow. of. closures. For h\y given reporting period, a low 
percentage — low, tiiat is, in relation. to other baseline figvrres — 
indicates a .service .flow that is too slow, and p.erhaps in need of adjust--^ 

mant. ' . ^ - ^' , ' ' . ' 

The results of this compaxisoA follow: 



Table 9 

Post-Acceptance Closure Rate ' 





' Percentage "of 


Agency 


Closed to Open Cases 


A 


■.' . 85% 


• 

""B 


69% 


C 


84% 



CFrom. the RSA-101) 

SfLsed on Agency A's comparison to the two other -agencies, the post^ ; 
acceptance closxire rate is fotjnd to be comparable to Agency C and only 
15 points above Agency B. In the absence of other baseline information, 
the manager rejects the service bottleneck hypothesis* The manager would 
then^examine past data on the number of .applic^ts 'desiririg VR iservices 
.to see if 'a recent influx of applicants is skewing the ttsnlts to mal^e it 
appear that expenditure/closure is prpblemmatic. fin the pretest analysis, 
we did not have the historical'/data to make this examination;) l'£ this 
does not explain the high expenditures per closure, then the programf^ 
-manager would assxime it to b^ a cost problem and inv-^sti^at? from that* • 
angle-, • » ■ . ^ 



S econd Level Indicator far Investigating High ExpendittTce ger Case 

The next situation to be examined is Agency A's high expenditure J)9r" 
casp. It is possible that 'the high expenditure « per rehabilitatiort^may be'* 
the result of an intake bottleneck.-- that is, the number of the cJokires 



is low because the agency is, accepting too few clients into service, thtis 
cutting off th* flow of potential 26 closxires.* ' ^ , . 

, As the first test of this hypothesis,* the ^manager . lopkV^t the agency 
rate of acceptance, or the ratio of newly accepted clients td all new 
applicants, on hand applicants, and clients in extended evaluation.* 

. i . Table 10. ' 
Acceptance Rate' 



t 


' Agency 


Acceptance- Rate .Cin %) 


A 


X. ■ ■ 39%' . 




B 


53% 




C 


43% . 

O. (r 



;/^31ie acceptance- rate (newly accepted clients is a percentage of all 
clients) agai^x shows . Agency A with low perforinaiice. However, before con- 
cluding that Agency A has- an 'intake bottleneck,, the prograa m^ger will 
also want to "exaaine the accepted VR pppxilation relative- to the state's 
populatioa (maaber served per 100,000 population, standard 1, data element 
ii)I This* investigation iweals the following: ■ - ^ • 



. ' Table 11 ' , ^' 

Clients .Per 100 > OOP State Population 



Agency 


, Number o£ 'Clients/100,000 


A 


■ '• -404^ • 




• ■ 6^0.6- 


C. ' 


, .474'.1- 





Based on- a low rate of .acceptance and a Idw numbet served per 100,000 

\- , ' . ^ ' " V\' 'i * 

population, the program manager concludes the existence of 'to in:take bottle 

neck, and calls for an examinati^ of .the- intake process, j^p' 
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Third Level Indicators ^d' ''Leading Questions" (see Table 6, ''Third 
Level Indicatbrs"}^^ ^^ * ' j 

At the third levei, various explanations of the int*ke problem are 
explored. Three Separate pra.ctices,. in combination or together, may explain 
why there -are too ;few clients entering the system: 

• emended evaluation is overused; > • 

• too many ineligihle clients are accepted; ^ 
too few people apply. ^ 

To determiiie whether extended evaluation, is overused, the manager 
examxnes the peiWitage of agency clients which are placed in extended 
evaiiiatidn (Status 06). In our example. Agencies B and C continue to be 
used for conq)arison purposes. Altema.tively, the agency might use- a pre- 
set sta n dard based on national norms or past state 'statistics. 



. Table 12 
Percentage of Clients in Status 06 



'ARcncy 


% 


A 




16.0* 


,B 




1.2% 


C - 




, 6. "5% 



(From, the RSA-IOJ.) 



^ The results indicate an unusually high percentage of clients placed , 
in extended evaluation. ' (T^is might be investigated further by obtaining 
data on -the average time in statu^-for 06 clients and reviewing the his- 
torical usa of status 0^ in Agency A and the. types of disabilities*' cbamon 
tA 06s.) The available figures seen to indicate an overuse of status 06. . 
However, the knowledge "of program managers may indicate that this is not 
the' case that evaluation research ,i,s required to . answer the problem. 

In addition to\the findings concerning the use of extended .evaluation, 
•the manager wiil kiso want. to, look at the perdentage of ineli-gible appli- 
cants. This .investigation reveals: . , - f 

' . '1 • ■ ■ . 

"^L . . . ^ \ • ^ ;/ 
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Table 13 

Percentage of Ineligible Applicants 



Agency 


% 


A ^ 


39.3% 


B 


31.8% . 


C 


41.6% 



(From the RSA-101) 

Based on these figures, the percentage of clients declared ineligible 
for services does not a^jpear unusual in Agency- A and is not judged to be 
a source of the intake probleni. ^ 

The last area to ]}e investigated in the effort to locate the source 
of the intake bottleneck is the number of applicants for VR services. For 
the purposes of example we will use* the state population as a base to allow 
for an across-states comparison. (If this were a within-state comparison 
of administrative r«?gions, the state can use local census data as the^base 
for calculations.) 

^ . y 

Table 14 

Applicants as a Percentage of State Population, " 







State 




Agency 


Applicants 


Population 


% 


A 


2,S10' 


"» 382,000 • 


■ .43% 


B. 


62,627 


^ 11,731-, 000 


-. .33% 


C - 


25,419 ' 


5,197,000 


' :'4S% ■ 




(Froja the RSX;-10l and Census Projections) 

Based on these results, the following conclusion^ can be drawn con- 
ceriiing the intake process in Agency A: 

^ The extended evaluation status seems to be overused,, thus 
reducing the pverall acceptance rate* Average time* in this'-^ 
status also n^eds tc^ be investigated. If this turns out to 

*•■•••■"■■■ 90 ,, . 
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be a problem, then the manager will want to do a within- state 
examination of practice, using the same table abalysis to 
compare administration regions, and to examine the types of 
clients that are being placed in 06, as well as the kinds of 
. services that are being planned. ^ 

• The uianber of clients declared ineligible for servi-ces do'es 
not appear to be a' soiirce of the intake problem* ~ - 

» With the available data the number of applicants has not been 
accepted or rejected, as a source of the problem; numbers of 
^applicants in the, past years should be examined to determine 
if this ratio is stabl^^/or is now particularly low. Also,^ 
an examination of the applicant pattern within the state may 
identify trouble ofdEices or districts. If the current number 
of applicants appears to also be a problem, then the program 
manager will want* to examine outreach and intake methods to ^ 
see if, they can be made* more effective. 

EXAMELELtiya: A CLIENT-LEVEL DATA ELEMENT, PERCENT 26 CL^J^S COMPETIT/ELY 
EMPLOYED ' ' • 

—^—^ ^ < 

In this discussion we pres;eht another example of the process to be 
used b^program managers in inv|sti gating problematic ♦ attainment, this 
tihe using a client-level data element instead bt an agency-level measure. 
As in Example One, we use comparative state-level data^ The s^e^ type of 
analysis can be used to determine patterns of perfprmance within states • 
(District, Region, Area, or t)f f ice comparisons.) 

■ ' . " ■ - ^ ^ • 

The Problem . ■ ■ 

For the example, - let us asstane that for data element SCi), percent 
26 dositres competitively employed, a goal of three such placements in 
four rehabilitations (75%)' has been, established- In the results of data 
element SCi), three of .the six pretest states showed prtblematic attain- 
ment- These states./ X, Y, knd 2, havi5 741,/ 73. 3*,' and 42.2% of 26 closures 
competitively employed, respect ivpi/. Agencies X and Y are general or 
combined agencies. Agency Z sexltes the blind. Ordinarily, Agency Z would 




be compared with other similar programs and perhaps would have a different 
"standard/' But for illustrative purposes, these "three agencies are/exam- 
ined together. . ' ' * 



First Level Indicators ' (see Appendix A, Standard ,Data Element 1) 

The potential ^problem identified is that a significantly large pro- 
portion, of 26- closures are not being placed into j;orapetitive erap^'loyrnent . 
With this in mind, tjxe first question asked by the program manager is 
''^fh at, happened to those non-competitively emprloyed 26 closures^?" To 
answer this, the program manager goes to the R-300's for 26 closirr^s and 
select^ for' work status at closure.^ 

This investigation reveals the following information; 



SIXTHS and 




TafaleJS 

Categories of *Mon-CCTpetitive Plac^ents 
as ji Percentage of Total Placement 





X 


Y 


• Z 


Sheltered Workshops 


11.3% 


0.'7% 


11.1% 


Business Enterprise V 




t 


2.2% 


Program 






Homemakers . • • 


13.6% 


26.0% 


43.3% 


Unpaid Fapiily W^kers 


1.1% 




1.1% 


TOTAL 


26.0%/ 


'^6.7% 


57.7% 



(Available on the R-300) 



' Of cotjrse, this information cannot ^e inte^rpT^ted in4ependentiy; it 
needs a qualifier to piit it in, the nec^sary Context. Once the ^>rogram 
manager knows the placement pattern -fo^ the non-competitively employed 26 
closures, the next question to be answered is "Ar6 these non-competitive 
closures appropriate?" More important, ''Is ^his'what the clients wanted?'' 
To answer this, the^ progragj .manager/again go'e^ to the R-300 's for 26 
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^Closures, and gets vocational goals vs. outcome. information (data element 
S(iii)). 



The findings from this investigatipn follows: ^ 



Table 16 



cJ 



Comparisons of- Work Status Objectives afad .Outcomes 







Y ■ 


Z 


WORK STATUS- OBJECTIVE • 

Competitive Gpal • • , 
Non-Conrpetitive Goal , " 


\ 

80.5% 
19.5% 


76.0% 
24.0% 


45.5% 
54.5% 


WORK STATUS^ OUTCOME 
. Competitive Outcome 
Mon-Competitive Outcome 


74.4% 
2S;6% 


73.3% 
26.7% 


42 .0% 

58.0% 

- / 

38,6% 

3.4% 
51.1% 


9BJE(!TIVE AND- OUtCCME ! / 

Competitive Goal - Competitive Outcome 
Competxtive Goal - Non-Competitive Outcome 
Non-Competitive Goal - Competitive Outcome 
Non-Coopetitive,Goal -' ^on-Compe.titiye' Qutcom*^ 


■ 71.0% 
9.5% 
' 3.4% 
16^% 


73.0% 
3.0% 
0.3%- 

23.7%- 



Witlj.<his information* being \xA^ in conjunction witii the work status 
at. closure '(^ee previous table), the prdgxam manager is now^in a position 
to mak4? some,.j£:o.nclusions about agency performance. 

On the surface, for Agency X, the "ynder-achievement'/ factor (competi- 
^tive goal and non-competitive outcome) seeijS to be. very high (9.5%); . 
But also notice that Agency X has tlw highesj: percentage of clients 
desiring competitive emplayraent as their vocational goal. In the absence 
of informatipn of past performance Cahd performances of other states) , 
Agency X seems to have encouraged mSny of their clients to strive for com- 
pericive'empl'dynjent. IVhile a large percentage of clients desiring com- 
petitive employment did not achieve this (9.5%), Agency X seems to have 
made an ^fort to maximize th^ir rate of competitively employed 26 closures 
Of those na&^ompetitively emp^py^^d^^^ey are Almost evenly distribui^ed 
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between sheltered workshop employees and homemakers. This would indicate 
that there is little or no creaming of clients into the homemaker status. 
Based on these considerations and the closeness of the , performance (74%) 
to the norm, Agency X's attainment on this data element does not appear 
problematic. 

Agency Y shows a high corirelation between'vocational goal and outcome. 

« 

The "overachievers" (clients with nonrcompetitive goals and competitive 
outcomes) and "underachievers" (cOTpetitive goals and non-competitive 
outcomes) have been minimized. Upon seeing the high correlation between 
goal and outcome, the program manager would want to examine this for other 
similar states as well. The manager might find that few states have such, 
a high goal-outcome correlationship, since it is perfectly acceptable for 
clients' and counselors' views on an appropriate vocational goal to change 
as the-pr^gram progresses. If this investigation reveals a potential prob- 
lem, th/ maiager might want 'to go to another step and examine • expenditure 
per closure (as ui the previous example) to see if the effort is being made 
to persuade clients to strive for a higher goal. Another area that may 
be problematic is the use of the homemaker status. We find that practic- 
ally all of the non-competitiVe closures were closed as homemakers (26% 
homemakers out of 26.7% nbn-competitively employed). As such, some 
(hypothetical) recommendation^for action for Agency Y are possible: 

1. Persuade counselors anjl clients to strive for a higher closure, 

status than homemakeir; Use performance measures to reward. 

"higher" closures and provide incentive for vocational 
/placement. ' I ' ' . 

2/ Direct"^ counselors /ot to overuse the homemaker status, but to 

use the 6ther noij/competitive statuses, or try for competitive 

employment . 

Agency Z shows Jh4 highest percentage of clients desi:^^iig non- 
competitive employment. As the sole blind agency in this knalysis, this 
agency's figures differ from^^the otlfer two. As previously mentioriedT^uch^ 
comparisons should ordinarily be made wi,th states. Over 50% of Agency 
Z'^ 26 closures strived for and achieved non-competitive employment. Most 
of these were homemakers (43.3%),' or employees in sheltered workshops (11.1%). 

" • - . 94 ■ ■ ■ , 
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For a blind agency, is a figure of 42% appropriate for competitive employ- 
ment, or too low? ' CAt this point, the program manager would want to compare 
the percentage of competitively employed 26 closures in Agency Z to another 
similar blind agency. 'In. our pretest, there were no other bli^d agencies* 
But, for the purposes of this exercise, let us assume that 42% is too law 
and that the result could be improved. Also, exogenous factors in Agency 
Z might influence the possibility for competitive employment. All these 
factors would influence an actual investigation.) Recommendations (hypo- 
thetical) for Agency Z for action include: 

1. Provide incentives for counselors and clients to strive for 
^ competitive employment placements . 

2. Initiate a job development program to stimulate the creation 
of plajcemint opportunities in competitive settings. 

Thus, in this partioilar example, one state agency does not, in fact, 
appear to have a "problem" on this element. For the other two, specific 
actions can be formulated • In the example, we used an arbitrary -perform- 
ance level to flag problems. A?i agency might adopt such a level as a 
policy, and set annual goals, or use previous perform^^e as a benchmark 
(use- the previous level, if it is acceptable, or plan for aa increase, if 
previous levels were too low) ^ * ' 

In both examples, we see how simple program measures, used in juxta- 
position -with, other^ measures and^ program knowledge, can be used to identify 
particular system components that may need attention. By themselves, most 
program measures give an incomplete picture at best, or may be confusing. 
But used as part of a step-by-step logical examination of progranr^erform- 
^ce, these measures can provide program insight. Appendix A of thisr report 
.'provides tables, like Table 6 in tliis "chapter, for other performance stand- ^ 
' ards data elements, plus a step-by-step guide to "th?. logical sequpiice of 
(lata comparisons. This Appendix is intended to be^used.-as a" guide to the 
kinds of analysis" described in this chapter. An agency may introduce other 
data and program information which ts available to support and enrich the - 
analysis . * ' : 

Som^ of the ''branches" in the lo^ic trees for the data elements sug- 
gest that further statistical work may J?e necessary. In* our model, sucn 




analysis is only suggested after several branches have been exploded, and. 
a nUaberof hypo]^eses ^rejected. There are 4^ number of statistical tech-|^ 
niques which have been developed in program' evaluation research. * 

EVALUATION RESEARCH ' ' ' y\ 

*In investigating problematic attainment there will be times when ^ 
further statistical analysis will be required, to -identify possible problems 
and corrective actions • Data sources for the statistical analysis in such 
evaluation research will be many and varied. First are those .sources, 
resulting . from routine 'reporting within the program; these include: 
r the R-300; , - ' • 

^ case reviews; . • j- ' 

• .closure and follow-up surveys; 

» the agency-level standard statistical reporting forms; 
^ caseload statistics; ; 

m summari^ of agency organization, types, resources, intetnal 

procedures,, and service provision patterns; and 
m the MIS and FMIS, 

In afl^tioTt to routine program reporting, other important data sources 
include iSoformation from other federal agencies and departments, and 
special studies conducted by RSA. or by contract research. For'^xample, 
the^^Pipartments of Labor and Commerce may provide useful information on 
national econou^ic trends and labor market conditions. And special studies 
may be conducted for several reasons: to generate new knowledge on var- 
iables hypothesized to impact on program success; to further study or 
validate the relationship between program success and independent factors 
that already have been observed to impact on program success;' or to update 
and/or provide informatiffn needed to test a "full" causal mod^l of program 
success (example here include the, needs for data on client motivation 
'and "counselor effort" by client). 

For the most part, agency- level evaluation research will rely on 
already-published 'data, usually based on the full population of VR 
clients. Data sources here include the program data boo^ and information 
on client characteristics and caseload statistics. Aggregate data on 
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performance will -436 merged wi tlx' information froitf- -other program reports 
(e.g., on costs, aumbers served, services offered, and case reviews) 
and from data received from non-RSA sources Ce.g., on economic and labor 
market trends) . This pool of aggregated' data will proVide the bulk 
>of infomation used in the agency-level evaluition research. 

Evaluation research may require some analysis that, calls for client 
data that is not regularly collected. If this is the case, such addi- 
tional data collection chould be done on a smaller sample of clients. 
Part of the data collection for the standards data elements is already 
-ifc. based on samples of clients, e.g., the closure survey, the case review. 
New data collection to support a client- level analysis would be «3n a one- 
time-only, is-needed basis and not part of the data collection system. 
Evaljiation research is intended to supplement the reports of state 
. * VR agency attainment on .the standards. After such reports have been sub- 
mitted. Vhey will undergo review by program evaluators and managers, who. 
will make preliminary determinatj:on^ of the necessity for evaluation 

research. . - . 

The most important question for the evaluation, research component 

concerns the types and levels of analysis for the research. First/, two 

types of data collection and of analysis are noted. Then, tWo iWels of 

' ■ statistical data anadysis, micro and macro, operational studies/ and 

statistical data studies, are described. ' 

/ 

Types of Data Collection and of Analysis: Ooerational/ S tudies vs 

Statistical Data Studies 

* So far all of the data collection that has bee.n described has been 
of a "quantitative?' nature, which is usually associated with evaluation 
I , research. However, there is a whole other style of data collection 
that is useful in evaluation research that is more qualitative. No 
attempt will be made here to specify the rayriad-^ types of qualitative 
data collection ~ there is. a vast Hterature on the subject (see e.g., 

/-■ Cook and Reichardt, or Bogdan and Taylor, or- Doug] as). The techniques 
of participant observation, of unstructured interviewing, or of in- 
vestigative social research could all- be usW to 'identify possible ^ 
problems and corrective actions. In the VR field, there is already a 
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structure rot 5uch data collection, around the PARs c>v SMARs/ The point 
is, that in the face of increasing data processing culpabilities., the. very 
important role .of qualitative information nnist not be 'lost. 

These operational studies can be used to generate hypotheses to *oe 
tested via>^ statistical data analyses,, or can be usedN^o understand the 
findings from the statistical .data analyses. Some of the qualitative 
iijfomation collected in the operational studies could be turned into 
quantitative *data, usually at the. ordinal or nominal level of measurement; 

Levels of Statistical Data Analysis 

One of the first choices that must be made -in deciding on th^ kinds^.- 
of statistical data, analysis to dp is the choice of a level of^alysis^^ 
Once the n^ajor choice between a micro and macro level ^f analysis is made, 
the analyst chooses the unit of analysis. Below, the differei^ce between 
micro and macro analysis is described.. Then, the relative advantages and 
disadvantages of a macro level of analysis are presented. 

Mitro 3nd Macro 
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The difference between a micro and a macro level of analysis is 
primtoily a ftmction. of the perspective, of the analyst i In economics, 
a miq^ level of analysis for understanding consumption focuses on the 
individual consuoerj' however, a micro level of analysis for unders t a n d in g 
production focuses dn the firm, a much, lai^ger unit of analysis. While' 
one analyst's micro level of analysis ^may be another analyst's macro level 
of axialysis, a crude but simple across-the-board distinction is helpful. 
Simply^ 2L micro level of analysis is at the level of the smallest pos- 
sible unit of analysis, with a macro level of analysis at some larger • 
unit of analysis, subsuming many of the micro units of analysis. 

' For social service delivery systems, at the micro level of analysis 
the unit of ajjalysis is usually the individual client. At the macro level 
of analysis tne. unit of analysis could be the counselor, the office, the 
district or region, or the state VR agency; in' each case several micro 
units of^alysis, i.e., clients, are subsumed under the macro unit of 
analysis . . . ^ ^ — ^ 

' as 
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A' macro level of analysis for understanding social service delivery 
systems^ has sevexal definite advantages over a micro level of analysis: 

y 1* Certain planning aijd policy* questions frequently asked are macro 
questions. The attempt^o set standards to guide the performance of 
state VR agencies ^is a macro concern; the question is not whether indivi- 

^ dual clients are receiving quality services, etc,, but whether the agency 
as a whole performs wall. . 

2* Given the short length of time clients are in most social service 

, delivery systems, changes in the system over longer periods of time cannot . 
be analyzed using micro data. Even in a year-to-year analysis, there are 
also problems in linking data for a given client. Of coiirse, there is 
some possibility of panel analysis,, but this is difficult and expensive. 
As such, with macro data time series analysis becomes a possibility. 

' S« Ihe measuremrat of a macro phenomenon can be fundamentally dif- I 
ferent from that of a micro phenoirienon. Here, the advantage of a macro 
analjysis is that the contextual effects of a particular program with 
speciAc organizational structure, clients, and services can be assayed. 
Two clients in different progow^ may have the same characterirtics and. 
receive the same services; however, the organizational stru ctur e and the 
mix of other servicfes and clients might result in a much different impact. 

4. Another advaxttage of a macro model is the aid in. the examination 
pf the effect, of •environment. A strong emphasis is placed in this analysis 
on the role' of environment as a limiting and an additi^re fqrce. It is very 
difficult to measure the environment at the individual client level^ Even 
if such mas^suremeht were possible,, there would be littj^ variation from 
client to client, especially for those served by the^same office or in the 
same local area.* For example, the unemployment rate is only measurable 
at a macro level. If assigned to each individual in the area or other 
macro definition, thW no variable would exist; there would be no varia- 
tion, thereby preventing any analysis, ^ 

'5. Irf a micro analysis individual differences and peculiarities 
come to the fore. In a macro analysis, these effects are wiped out. 
Neither is necessarily berper. Sometimes the eraphasis is on larger 
structural effects, and for this macro analysis is better. 
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\ ^6. Micro dat;a can always be aggregated, whereas disaggregation of 

macro data is often difficult sometimes impossible.^ Thus, micro is , 
often preferred 'to macro. However, in social service delivery systems 
some data are available at the macro level which are not available at the 
micro level. , - * 

♦ 

7. Certain concepts for understanding certain behaviors in asocial 
service delivery system are macro concepts* For example^ to understand 
client selection, the concept of population-at7risk is necessary, and 
this concept is a macro concept.^ 

Both macra and micro analysis are necessary for evaluation' r^s^arch 
for the data-based decision support system. For*VR evaluation, micro 
analysis refers to ciient-levei analysis, although some macro data* ct^d. 
be included in that analysis (e.g., attaching counselor characteristics 
to client information). Macro analysis refers to agency-level analysis. 
The focus of the federal standards system is on state VR agency behavior' 
and, for RSA or national reporting, the- macro level may be more usefiil. 
vMicro analysis of subunits within the states, and of counselor behavior, ' 
will be more useful to individual state VR agencies. j 

Decision support is a term- that covers the activity of using standards • 
infoxmation md^other program information to answer questions about the 
state's attainment in the provision of rehabilitation services. Through 
the creative. use of this appifoach to employing program ' information , man- 
agers can work in the identification of practices and environmental con- ^ 



ditions affecting atta:^nment. 
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VI THE PROGRAM STANDARDS EVALUATION- SYSTEM REPORTS 

In the proceeding chapters, we hmvh reviewed the standards -and their 
data element measures, and illustrated how thdse measures can be used to 
diagnose s^ency problei^ and point to corrective actions. To assist 
agencies in use of the standards s/Stem, we propose a series of routine 
reports' to show state performance on the elements, both internally and in 
relationship to ^S^t^sm^s. In addition, a sefbf designs for displays 
of related data items, useful for decisis?n support system analysis, is . . 
part of ^e reporting system. Both types of reports are intended to * 
clearly represent standards findings to agency management. The reports • - 
have been designed in formats for computer generation. 

The standards reporting system' brings together the various sources 
of standards input data so that a paxticujlar agency's attainment for a 
specific time period' can be compared to its objectives for the period. 
In addition^ the reporting system will provide the program, managers with 
the aqjability to flag, and investigate problematic 'attainment, as we- shall 
describe subsequently. To do these two things, the reporting system has^ 

been designed" to: . .' ^ . 

" » kfeep trade o£ past. performance as well as current expectations; 

• present the findings in an ea^ to use, easy to understand way, 
■ with^t unwieldly reports, emphasizing jpraphical presenta^tions 

as well as plain numbers; 

• make s^r^ that-, the repottilig of results . occurs in a timely 
f^hion, -so that futT5\performance caft be influenced. 

The standards system uses input data routinely generated even now in, many 
state agencies' 'internal information systems. Thus, the evaluation stand- 
ards /system could be readily a4apted by individual state agencies for their 
use! The calculation of national norms would require a national, system . 

howeveit-. e ' . ' ^ 

.Tables 17, 18, 19 , "and ZQHi^ustrate reports that can be routinely 
generated for ^e Performaiice Standards'. The first set of reports (one 
state's example is seen, in Table 17) shall show achievement on each of the 
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Table 17 \ ^ » 



ACHIEVEMENT. ON PERFORMANCE STANDARDS 

Year': 1981 
■ State: California 



This ^ 1980 
Thi 3 Years Nat ' l" 

Year Goal 1980 "Norfn 



1. COVERAGE 



(a) Clients* served per 100^000 
population ^ ^ XXX.X " XXX.X . XXX.X XXX.X 

(b) Percent severely disabled 

served * XX. XX XX^iXX XX. XX XX .XX 



■ [ 



2. COST-EFFECTIVENESS AMD BENEFIT 
C^ST RETURN 

<aJ Expenditures per compet- 
itively employed closure SXX.XXX SXX.XXX SXX.XXX SXX.XXX 

, <b) Expenditure per 26 (Ros- ' 
^ ure ' SXX.XXX SXX.XXX .SXX.XJCX SXX.XXX 

<c3 Ratio of total VR benefits 

to total VR costs ^ xx.xx -xx.xx; xx.xx XX. xx; 

<d) Total net benefit from; 

services- XXXX.X XXXX.X XXXX.X XXXX.X 

^ REHABIUTATION RATE / " 

* • (a) Percent 26 closures XX.XX XX.XX *XX.XX^ XX.XX 



<bJ Annual change in number ^ . , 

of 26 closures 'XXXX XXXX- XXXX ' ^ XXXX 



4. ECONOMI^ n^OEPENDENCE 

(a) Percent 26 closures with 
^ ' weekly earnings at/above 

fe^ral rhinimuo wage XX.XX ' » XXiXX XX.XX XX.XX 

•<b) Comparison of earnings of ^ 

competitively employed 26 ' * , 

closures to earnings of , ^ 

, ' employees in state * X.XX X.XX X.Xv X.XX 

/ ... , 

5. saiMrjl activity 

<a3 Percent 26 closures com-* , 

petitively empJLoyed^ XX.'XX XX«XX XX.XX XX.XX 

<b) Percent competitively 
^ employed 26 closures with ' * . 

hourly earnings at/above ' • 
O T , federal minimum wage XX.XX XX.XX • XX.XX XX.XX 



ERIC 



102 



95 



{ 1 

Table 17 (continued) ^ 'J 

Thfs 1980 

This Years NatH 

' Year . Soal 1980 Norm . 

i • . • , ' 

5. GAINFUL ACTIVm (continued) 

(c3 Percent noncompetftiveLy 

employed 26^Losures showing 

i-mprovement in functioning * „ w „v 

and Life status • / - XX.XX XX.XX XX.XX XX.XX 

6. CUENT CHANGE - „ " x 

(a) Comparison of earnings before 

and after VR services SXXXX.XX SXXXX.XX SXXXX.XX SXXXX.XX 

(b) Changes in other statuses 
and functioning ability 

7. RETENTION 

(a) Percent 26 closures retain- 
ing earnings at follow-up, 

(b) * Comparison of 26 closures 

with public assistance as 
primary source of support 
at closure and at .fol.low-up 

(c) . P^ercent noncompetit^vely 

ewployed 2-6 closures 
retaining closure skills^ 
at fol Low-up 

8. SATISFACTION- 

(a) Percent closed clients^ ^ 
satisfied with^ overall 
VR experience 

(b) Percent* closed clients 
satisfied with: 

m 

counselor 

physical restoration . 
job training services ( 
placement services 

(c) .Percent -26 closures judging 
^ services received as useful 

^in obtaining their j*ob/ . * * 

homemaker s^ltuation or in ' . . . , 
current perfdnKnce . XX.XX XX.XX XX.XX XX.XX 



XXX. X 


XXX. X- 


XXX. X. 


XXX. x' 


XX.XX' 


xx.xx 

>• 


xx.xx 


xx.xx 


xx.xx 


xx.xx 


xx.xx 


xx*xx 


xx.xx 

* 


V 

xx.xx 


xx.xx 


^ xx;xx 


xx.xx 


t 

xx.xx 


'xx.xx 


« 

xx.xx 


xx.xx 
xx.xx 
xx.xx 
xx.xx 


xx.xx ■ 
xx.xx 

XX .XX 

- xx.xx 


xx.xx 
xx.xx 
xx.xx 
xx.xx 


✓ 

. xx.xx 
xx.xx 
xx.xx 

XX;. XX 
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ACHIEVEMENT 



TafaLe 18 

ON PERFORMANCE STANDARDS t STATE COMPARISON 



GENERAL AND COMBINED AGENCIES 



Standard :J -vl* 
Data Element: <a) 



COVER|,GE 

Clients 'Served per 









100/000 Population 




* * 


National 


Norm: 


XXX.X 












This 
























AGENCY 


Year 


Goal 


1980 


1979 


1978 


1977 


At ARAMA 


AAA • A 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


Al A^l^i 


AA/\ • A 


XXX *j( 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


ADTTHMA 


XXX X 

AAA mA 


XXX ^ 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


' t 


XXX X 
AAA • A 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


' TAI TPflPMTA 


xxx.x 

AAA • A 


5?XX.X 


XXX.X ■ 


XXX.X 


XXX.X 


XXX.X 




XXX X 

AAA • A 


XXX.X 


XXX.X 


XXX .X 


XXX.X 


XXX.X 




XXX X * 

AAA • A 


XXX.X 


^ XXX.X 


XXX.X 


XXX.X 


XXX.X 


net AUAOC 


XXX X 

AAA • A • 


XXX.X 

AAA . f\ 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


r^TC^^ Off rnf hmota 


XXX X 

AAA • A 


' XXX.X 


XXX.X 


XXX.X- 


XXX.X 


XXX.X 


rUUnXUIr 


XXX X 
AAA •A 


XXX.X 

AAA • 


XXX.X 


XXX.X 


XXX.X 


XXX .X 


GEORGIA- 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


GUAM 


. XXX.X 


XXX.X 




XXX.X 


XXX.X 


XXX.X 


HAWAII 


XXX.X 


XXX.X 


XXX.X ' 


XXX.X 


XXX.X 


ItXJC.X 


IDAHO 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


'ILUNOIS ^ 


xxxlx 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


INDIANA - 


XXX.X 1 


XXX.X 


XXX.X 


XXX.X 


XXX.X, 


XXX.X 


IOWA ^ • 


^ XXX.X / 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


KANSAS 


XXX.X 


XXX rX 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


KENTUCKY 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


LOUISIANA 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


MAINE 

• 


_xxx.x 

• 


XXX.X 

• 


XXX.X 

• 


XXX.X 

• 


XXX.X 

• 


XXX.X 

• 


• 
• 


• 

7 

• 


• 
• 


• 
• 


• 


• 


• 

t 


• 


* • 


• 


• 


• 


{ . 


• 
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Table 18 (continued) 



BLIND AGENCIES 



AGENCY .. 

CONNECTICUT 

DELAWARE 

FLORIDA 
C IDAHO : 

IOWA 

KANSAS 

KENTUCKY 

MASSACHUSETTS 

MICHIGAN 

MINNESOTA 

MISSISSIPPI 

MISSOURI 
4 MONTANA 

NEBRASKA 

NB^ JERSEY 

NEU YORK 

NORTH CAROLINA 

OREGON 

PENNSYLVANIA ^ 
RHODE ISLAp, 
SOUTH CAROLINA- 
TENNESSEE 
TEJCAS 
UTAH 

VERMONT • 

VIRGINIA 

WASHINGTON 



This 
Year 

XXX. X 

XXX. X 

XXX. X 

XXX. X 

XXX. X 

XXX. X 

XXX .X 

xxx.x 

XXX. X 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 

XXX Jf~ 

xxx.x 
xxx.x 

XXX.X 

xxx.x 
xxxftx 
xxx.x 
xxx.x 
xxx.x 



This ^ 
Years 
Goa I 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
.xxx.x 
xxx.x 
xxx.x 
xxx.x 

XJiCX.X 

xxx.x 

xxxi-x 

XXX.X 

xxx.x 
xxx.x 
xxx.x 
XXX. :H 
xxx.x 
XXX .-x. 
xx^.x 



• 1980 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x A 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

•xxx.x. 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 



■ 12Z2 

X){X.X 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 

XXX J( 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 



1978 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
XXX Ix 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 

XXJf.X 

xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 
xxx.x 



1977 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x . 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

xxx.x 

XXJ^X 

xxx.x . 

XXX.X 

xxx.x 
xxx.x 
xxx.x 
xxx.x . 
\xxx.x ^ 
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. , Table 19 

ACHIEVEMENT ON PERFORMANCE STANDARDS; SUB-STATE COMPARISONS 



Standard: 
Data Element*. 

State Goal: ^ 
State Average: 



'1. ■ Coverage ^ 

(a) Clients Served per 
100,000 PdpuLatiort 

. XXX. X 

XXX. X 







This 






• 


DISTRICT 


"This 


Years 








(OR OFFICE) . 


Year 


Goal 


",1980 


197? 


1978 


^ ■ A 


XXX.X 


XXX.X 


. XXX.X 


XXX.X 


XXX . X 


' 8 . 


XXX. X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


C 


XXX.X 


XXX.X 


X)^X.X 


XXX :x 


XXX. )[ 


0 


XXX.X 


XXXcX 


XXX.X 


XXX.X 


XXX.X 


e 


XXX.X. 


XXX.X 


XXX.X 


xxxCx 


xxx*x 


F • 


XXX.X 


XXX.X 


-XXX.X 


XXX.X 


XXX.X 


6 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


H 


XXX.X 


XXX.X 


XXX.X 


XXX.X 


XXX.X 



99 



Table 2Q 

ACHIEVEMENT ON PERFORMANCE StANDARQS 
ALL AGENCIES . > 
Year:- 1981 . 



Nat • L Gen- Corn- 
No rm'^ eral ' Blind- bined 



•1. COVERAGE 



Ca) Clients served per 1QQ/Q00 ^ , 

population ^iXX^ XXX»X XXX. X XXX. X 

Cb) Percent severely disabled ' • 

seryed XX. XX XX. XX • XX. XX XX. XX 

2. COST-€FF€cTIVENESS AND BENEFIT 
COST RETURN 

(a) Expenditures per compete-' ' - ■ 

tiveLy ertpLoyed closure SXX.XXX SXX.XXX SXX,X}?X ' SXX.XXX 

(W Expenditure per 26 closure ; SXX.XXX SXX.XXX SXX.XXX SXX.XXX 

Cc) 'Patio of total VR benefits - ; 

to total VR costs /XX.XX XX.XX . XX. XX XX.XX 

. *Cd) Total net benefit from VR * * ' 

^ services ' XXXX.X XXXX.X XXXX»X XXXX.X 

3. REHABILITATION 

CaJ Percent 26 closures XX.XX XX.XX XX.^CX XX.XX 

^Cb) Annual change in number ^ 

of 26 closures ; XXXX XXXX^ XXXX XXXX 



4. ECONOMIC INDEPENDENCE 



Ca) Percent 26 closures with ' , 

weekly earnings at/ above 

■federal diinimum wage *^ XX.XX XX.XX XX.XX XX.XX 

Cb) Comparison of earnings of - 
^ competitively employed 26 - - ' 

closures to earnings of , 

employees in state X»XX X»XX X»XX, X'.JtX 

5. GAINFUL ACTIVITY * . - 

Ca) Percent closures com- 
petitively ^mployed XX.XX XX.XX XX;XX XX.XX 

^ Cb) Percent competitively 

employed 26 closures with . 

hourly earnings at/above , . • 

federal minimum wage - XX.XX XX.XX XX.XX XX.XX 
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• Table 2Q.*<c6Pitinued) ^ 



Nat • L Genr 



7; 



5, , GAINFUL 'ACTIVITY •C|ontinued) 

c'c3^JPercwt norfcompetitiveLy 
^ ^' _efflpLoyed 26 closures showing 
^.imppovelnent ^n functioniag 
; 'and Life status 

6. CUENT CHANGE 

(a^) * Com'paplsbft of earnings j ^ 
before and after VR ^ 
/services ^^""X^ * ' > 

Cb) Changes jn other statuses* 
aild functioning ability 



Com-* ' 

Norm^, er'al ^< Blind bin^d 



XX. XX 



XXiXX XX. XX XX. XX 

SXXXX.XX SXXXX.XX SXXXX.XX sxxxx.xx 
• XXX. X . 



XXX.^< XXX.X XXX.X 



RETENTION \ 

(a) Percent 26 closures retain^ 
img earnings at foLldw-njp 

(b) Comparison of 26 closures 
with public assistance as 
primary source of support 
at clo^iure and at follovnjp 

<c) Percent rioncompeti ti ve Ly 



XX.XX XX. XX. XX. XX XX .X 



XX.XX XX.XX . fCX.XX XX.XX 




esnpLoyed 26 closures 
retaining, plosure skills 
at follow-up 


• 

XX.XX 


•\ 

XX.XX 


* 

XX.XX 


XX.XX 


SATISFACTION 

(a) Percent closed Hiertts • 
satisfied with ovejrall 
VR experience 


XX.XX 


•* 

, XX.XX 


XX.XX 


XX.XX 


(b) Per ceirt^c losed' c Li ent s 
satis-ned withr . 


t 








counselor ^ 


XX.XX 


XX.XX 


XX.XX 


XX.XX 


physical restoration 


XX.XX 


XX.XX 


XX.XX 


XX.XX 


30b training services 


XX.XX 


XX.XX 


XX.XX 


. XX.XX 


. p lacemtf nt servi ces • 


XX.XX 


XX.XX 


XX.XX 

/ 


XX.XX 


Cc) Percent 26 cLosOres judging ^ 
services received as useful 
in obtaining their job/ 
homemaker" situation or in ; 

\ current performance 


^X}(.XX 


xx.')<x 


V XX . XX ^ 


1 

1 

^xx.xx 
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standards for a, given agency. ^ In addition to shewing this year's perform- 
aace, the table also will show. the state's goal for the yeaf, its last 
yeaf^ performance ,^ and tJhe previous ' year ' s national norm. /With this 
information, agencies can- see how. 'successful they were in meeting their 
goals', for each of ;the dat^ elements . They can also compare this year's 

iwrformance with last year '"s. to see wh4re they have.agd hav'e not iinproved. 
» finally, agen^ei.,can 'assess their curtent' performance in relation to 
recent national norms.' This ty^je of report gives program managers an over- 
all viAw 0^ agency performance while' at the same time pointing out specific 
strengths and weaicnesses/ currentjV and oT^r time. If problematic attain- 
ment is identified, the data based decision support system can be used to 
assist managers in daignosing and correcting the problem. A particular 
advantage of mpst of the system reports (Tables 17 and 19) is that th^ir 
"turnaround" time can.be relatively short because the reports use only the 
individual agency's data (and ,a previous year's national norm) . Computing 
' current year national or regional narms for Tables 18 and 20 requir^ data 
submissions fronj ail 1relpvap€'states^. 

In addition; national fepprts should be- prepared by RSA and/ or the 
Council of State Admini^tralfilrs 'of Vocational .Rehabilitation for each data 
element that wii;L display all agencies' performance on each particular 
•element., Table 18 shows an. exampl.e for data element l(i). ihxs year's 
^oalN^s >feH as performance in the four previous years can be presented. 
Agencies can .use 'the information to compare their perfoTTnance and their 
goals to other similar agencies. By providing dstta for the four previous 
years', trends over time can be analyzed. Agencies and RSA will be able to 

.de.teTmine if perfofnance has steadily improved over time or if this year's 
performance is noticeably different than previous years. States are listed 
in alphabetical order; they could easily be sorted and listed* by region, 
by type of agency, or by performance. Looking at the national data 'in 
Tabie 18, a state VR agency could even expand the national data output dis- 

" played in Table 17 if desired. ' _ ' / 

■ Table 19 has an output design like Talale 18, but shows information by 
sufastjate unit. (Regions, Areas, Districts, Offices, or Counselors can be 
used- in this breakdown, according to agency administrative structure and 
size.) This report' is useful for examining problem .areas withjn states, 
to account for problematic performance at the state level. 
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Table 20 shows an example report of national performahce for e^ch data 
element for all agencies, and for general, combined, and blind agencies. 
TJiis aifows a program-wide v^iew of performance in VR. 

These three types of reports can be generated routinely fipr all of 
the agencies and all of the data elements. They provide information in. 
a'brief, easy-to-read format^ In addition, RSA and the agencies will haye 
the capability to use the standards information to generate special puipoW' 
report?, analyses, and graphic displays/ For exansple,^ the basic reports 
could be run separately £&r special populations. These may take 'tiie form 
of statistical reports or of graphic disprlays • ' 

. The 'system calls for access to a number of ^uppoirting -inf ormation » 
items useful in analyzirig. and interpreting the- routine reports . The«e 
information items feed into the decision-support sy|taii discussed ^|arlier. 
Based on any problems thar.cmer^ in the agency's standards performance-, 
progiam managers will inspect particular information itemk keyed to the- 
various standards data elements. ^ / / 

The- displays, on: the ^llowinff pages are to be used to fbllw the 
dectstcrr tr ees step.-t^rw5tep; they illustrate possible displays^ for an 
int eracti ve coi^puter system* A complete ser o£ these displays, for- the- - 
- ^ g rCo n aa ace standards: data. ele&»nts,* be found witte the corresponding. 
'*3Sbcisxott- tree tables iir Appendix k o£ this r e po r t s - - 
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Figiire S 

m 

Example Displays of Steps in Analysis 
of Rehabilitation Problems 



DISPLAY 



, PRIMARY PROBLEM: THE PERCEmCE OF THE CASELOAD THAT .IS SEVERELY DISABLED 
' IS TOO LOW ■ " 



TEST 



3T LEVEL INDICATORS) 



# SEVERELY DISABLED IN CASELOAD 
XXXX CNOPM) .< 

Xm (VALUE) 

I 

A. TOO LOW? ■ • ' 
YES 
NO 



REHABILITATION RATES FOR SDs COMPARABLE 
TO AGENCY REH^ILITATION RATE? 



xm CNORM) 
XXX}{ CSD RATE) 

B. LOWER THAN AGENCY RATE? 
YES 
NO 



>7 



IF NO TO BOTH A. _AND B., GO TO DISPLAY 1.2.1- 

IF Nd TO A.. YES TO B., GO TO DISPLAY 1.2.2 

IF YES TO A., NO TOB., GO TO DISPLAY 1.2.3 

IF YES TO BOTH A. AND B., GO TO DISPLAYS 1.2.2 AND 1.2.3 
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Figure S (continued) 



DISPUY 1.2.1 

K , . - • • 

SECONDARY PROBLEM: THE PERCENTAGE OF THE CASELOAD THAT IS SEVERELY DISABLED = 

IS TOO LOW^ BUT THE yNUMBER OF SEVERELY DISABLED CLIENTS 
•MEETS AGENCY STANDAROS AND THE REHABILITATION BXTE OF ' 
THESE CLIENTS ^S CLOSB TO THE OVERALL AGENCY RATE OR BETTER 

TEST (SECOND- LEVEL INDICATOR): DOES THIS REPRESENT EXCESSIVE COSTS? 
COST/CLOSURE (D.E. 211): 



I 



TOO HIGH? -YES- 
. NO 



A. ANALYZE COSTS rs.D., NON-S.D. TO SEE IF COSTS ARE- COMPARABLE. 
IF NOT: 



B. (BCAfllNE S.D. CASES IN CASE REVIEW TQ DETERMINE IF TOO "WU* IS SPENT 
^ ON. THE CASES. ' ' - 

= IF -YES; 

C". GO TO DISPLAY 2.2.0 ^ ' '> 

IF. NO, LOW PERFORMANCE ON THIS DATA ELEMENT DOBS NOT INDICATE A PR0BLE«1 
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Figure S (continued)/ 



DISPUY 1.2-. 2 



SECONDARY PROBLEM: THE PERCENTAGE OF tHE CASELO^ THAT IS SEVERELY, DISABLED IS 

TOO LOW, THfi NUMBER OF S.D. CLIENTS MKTS AGENCY STANDARDS, 
* BUT THEIR REHABILITATION RA^ IS LOWER THAN THE. OVERALL 



AGENCY * RATE 

. . f 



TEST (SECOND LEVEL INDICATOR) : EXAMINE THE RATE AND TlteLINESS FOR SIJi CLIENTS 



TIME IN PROCESS 

Xm . (SDs) ' ' 

XXXX OiOm: 

LONGER TLME FOR SERVICE? <- 
YES ■ 

« 

NO " ; . " 



r 



IF YES, FORECAST CLOSURE DATES AND ANALYZE FUTURE COMPARABLE ftATES TO 
TEST FOR LONG-RUN STABILITY ON THIS ELEMENT ■ 

ANALYZE NaN-SUCCESSFUL SDs TO DETERMIJ^E REASONS FOR LOWER REHABILITATION 
RATE , \ ^ • • 
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"Figure S v( continued} 

DISPLAY 1.2.3 



SECONDARY PROBLEM: THE PERCENTAGE »0F THE "CASELOAD THAT it SEVERELY DISABLED IS 

. _ TOO LOW, THE NUMBER OF S.D. CLIENTS IS TOO LOW; THdSE CLIENTS 
" DO, HOWEVER, HAVE A SUCCESS RATE CLOSE TO THE'AGENCY RAT? OR- 
BETTER ;~ 



TEST (SECOND LEVEL INDICATORS) : TOE PROBLEM IS EITHER INFLOW APPLICATION OF 

SDs, OR IN THE ACCE?TANCE RATE, OR BOTH 

» 02s SD CVALUE) XXXX 

i"02s ' ^ 

(NORM) XXXX 



JOO LOW? 
YES 
NO 



1" 

IF * -023 SD IS TOO LOW, REVIEW "oUTREACH PROCEDURES TO INCREASE NUMBER OF 
ELIGIBLE SD APPLICANTS. 



% CLOSED OS BY REASON- OF SEVERITY: 
' (VALUE) XXXX 

(NORM), XXXX ^ ^ 

1 

\ 

TOO HIGH? ■ / 
YES 
NO 

IF YES, DOiCASE REVIEWS TO EXAMINE THE CASES aOSED TOO SEVERE. TO 
ASCERTAIN 1^ CRITERIA FOR ACCEPTANCE ARE TOO LIMITED. • 
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*VII. PROGRAM MANAGEMENT AND TOEJJSE OF THE STANDA^ SYSTBl . .V/ 

' In this final chapter, the role the standards "system will play in the 
^VR system is discussed, with an .example of the tise of the st&ndards system. 
The actors who will be taking .the correction actions are identified. The 
need for evalioating and changing the standards syste^ over xime is discussed. 
Finally, some comments on implementing the standards system are made. 

A REVIEW QF^-mE PROGRAM MANAGERS ' USE Of THE STANDARDS SYSTEM • ^ 

The steps in manageaent^use are: / 
,(1) RSA and state VR agencies set objectives on ^ach of the stan- 
^ dards data elOT^ts for the cycle of operation; f 

(2) Reporting system flags problematic attainment bys^ particular 

state^VR agency on a particular data element; • • ► 

(33 RSA program managers and state VR agenty program manager using 

the decision support ^system identify a possible problem and ^ . 

co r re c t i ve actions ; and ^ ^ '/ ^ 

•C4) Correction actions, ar^ taken by the various actors in the VR 

system. 

'An abbreviated example of the progjram managers' use of the standards 
systea^ follows* Consider the clear vopational thrust of /the VR program. 
While there are several manifestations of this thrust, o^tej^ortant goal 
for VR is econoBdc indep'endence for the disabled client. Thxs goal is ^ 
represented in the revised standards by Performance Standard 4. 
The data elements for this standard are based on the .measurement of weekly 
eamingsl at closure • Siqjpos^ the reporting. systjMn identified a State 
agency w,th a low level of achievement for this standafi, compared to its 
performaice. expectation. The data basdd decision/ supper^ system would be 
used to identify why the state agency had this level of attainment and make 
recomm.endations about how to improve tho^ agency's performance. First, — 
^ program managers would try to, identify problems using the information in 
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the standards teportihg system and the logic o£. the decision support sys- 
tem. The analysis may readily identify the action steps necessary to im- 
prove petformance, or, more statistical analysis may be necessary. Vsing 
thT macro (agency- specific) analysis, we might find thait, after controlling 
for other factors, what explains low performance on earnings is the number 
of severely disabled in the caseload. However, no ^action" tor management 
is evident. Certainly, one would not recommend servin^feWer severely 
disabled, given the priority for them in the 1973 legislation. More^ 
evaluation, in the form of client-level analysis, might identify that cer- 
tain services are .mote useful in combination for obtaining hi^er paying 
jobs for the severely disabled. Possible corrective actions include: 'the 
issuance of an Information Memorandum on the usefulness o'f certain ser vices 
for the severely disabled, perhaps giving *such services priority; tecfanii^al 
assistance by the regional office to help state agencies in the prqvisi<ai 
of^.such services;, traiijing of counstfiors by state agencies the use 
these services; funding of researdi, evaluation, of demonstrations by RSA 
fbr service provision^ to the severely disabled, thereby involving the Depart 
meiie o£ Education, 0MB, and Congress; and promulgation of new regulations 
^by RSA if needed. j 
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ACTORS AND CORRECTTVE ACTIONS . 

~ ^ ^~ j • : 

The actors with responsibility for making changes in the standards 
systeii are the same as in the VR systCT at large: Congress, 0MB, Depart- 
ment o£ Education, RSA, Regional Offices of Rehabilitation Services, state 
governments, and state VR agencies. The set of actbrs and associated types 
of corrective actions are g^en below. 

. Actors ^ Types of Corrective Actions 



Congress, 0MB, Department of Funding levels 

Education * Allocation formulas 

Priorities to client groups 
^ ' ^ ^ Procedural" reqtii'rements 

RSA Regulations 

Monitoring 
Evaluation 

Research (alonj with NIHR) 
^ Program development 
Guidance materials 
Training programs 
Demonstr at i ons 
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Regional Offices 



State Governments 
State VR Agencie's 



.Technical ^assistance to state 

VR agencies 
Dissemination of information 
Diffusion of innovations 
Training 

Funding levels 

Same is RSA- (e.g., regulations, 

evaluation) 
Eligibility determination changes 
Counselor training 
Case management changes 
Service provision changes 
Management of sub-units (e.g., 

districts, offices) 



EVALUATING THE STANDARDS SYSTEM OVER HME , 

The criteria for evaluating, the revised standards system are very 
simple. The most important evaluative criterion is whether the attainment 
of the' state VR agencies is improving in the areas measured by^ the stan-^ 
dards data fl^lement^; While it may be very difficult ^o prove that the 
cause- of the improvement wafe. the implemeatation of the standards, at 
least the attain^pent of the agencies after the implementation can be 
comp ared to their attainment before the implementation- The secon^ eval- 
uation criterion is whether the state VR agencies are meeting their objec- 
tives . If they never meet their objectives, then the objective setting 
process is not working properly. I^f they ^ways meet their objectives, 
. then the process is also not working pro^perly. Identifying for which 
state VR agencies, for which- data elements, or for both in combination, 
which objectives are not being met, will indicate where attention needs 
to be paid in the standards system. The third evaluative criterion is 
whether the program managers find the system useful . Pro'gram managers 
should.be regularly canvassed for their recommendations. 



CHANGING THE STANDARDS SYSTEM 



)iUty. aI t 



A key word for the standards' system should be flexibility^ A* the 
standards system operates, feveral factors outside the system may <|hange: ^ 
the goals and functions of the VR program may change, necessitating 
changes in the standards; 
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• reporting ^ within or without VR may change, changing what will 
be availabl]^ for the reporting system; 

• the actors and jfpes of corrective actions possibly may change; 
actions taken by state VR agencies might push the VR program 

in undesirable directions, as^ state program managers try to - 
respond to the standards system, thus requiring additional / 
standards or changed expectations; and 

• the achievement of the statSpVR agencies may not be improving 
^ over time. 

t / . ^ 

^A^numfaer of factors inside the system may need change: , 

t some data elements may be found to have lower data quality than 
is acceptable, and thus require new procedures or even replacement; 

• some of the data collectioij activities may require change, because 
of logistical problems; 

m difficulties in- the reporting system and^-in the reporting cycle 

^maV arise; and 
» objective^ being set may not be correct. 

As such, RSA must monitor the' operation of the standards system over 
time.- In the beginning, the system should especially be closely monitored, 
so that proljlems can be discovered early. RSA must be ready to change tjie 
data elements in the standards system as needed. 

PtPLEMENTlNG THE STANDARDS SYSTEM ^ 

In its design of a standards system an^ in this presentation of the 

analytic paradigm, BPA has tried to : 

clearly identify the benefits of the use of the standards 

to the state agencies and to RSA staff r v 

• make the presentation of the paradigm as clear as possible; and 
^ keep eventual utilization in mind throughout. 

To implement this system, state VR agencies and RSA shoiild involve, from 
the yery beginning, those program managers and others who will have to act 
on the standards state or RSA staff, agencies or regions, the data com- 
raittee of CSAVR, and other users of the standard^. State VR agencies and 
RSA should also ''sell" the standards system, through widespread promulgation, 
and agency endorsement, llS 
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APPENDIX B * 
DIMENSIONS OF A METOOD FOR SETTING CfBJECTIVES 

Implicit in methods for setting objectives are many different choices 
to /be made in designing a method. These various choices are characterized 
below as dimensions of a method of setting objectives. In effect, a com- • 
bination of choices along all of the dimensions defines a partictilar method 
Given the number of dimensions and gradations along each, a very large num- 
ber of methods can be generated. Rather than to be used to identify all 
• possible methods, the discussion below is to be used to delineate the many 
^ issues around the final decisions on a method to be recommended. 

The dimensions along which a method for setting objectives can differ- 

include: ' ■ , 

•> general vs. agency-specific objectives'; . - . 

• technical vs. noznative approach; ^ 
m wytn-ttmnB vs» TwrfTiHTin objectiv^; 

logical vs* no logical basis for level; 
^ historically vs* ahistorically derived objectives; 
^ ^ nationil vs. sub«national coinparisoxts ; 

objectives for agency vs. for sub-agency? 

adjtxsted vs« unadjusted objectives; ' 
» singl* data eleaent objectives vs^ inc or p o r a ting relationships 

into objective setting; 
» stipulated vs. negotiated objectives; and 

• general vs. standard speci^c methods. 

General vs. Agency-Soecifie Objectives 

The simplest dlmeflsion to consider is whether to. set these objectives 
for all agencies ar once, or whether to tailor the objective to individual • 
. . VR agenci«. While .for certain standards general levels of attainment 
• might makl sense, for example in the area of corapiiance with the IWRP re- 
quirementk, for most standards an agency by agency setting; of objectives 

ErJc ; ■' . 125 • ... 
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seems necessary. The differences between agencie^ are great enough to war- 
rant this individual attention in the objective setting process. 

I 

Technical vs. Normative 

One of the most inportant dimensions along which methods of- setting 
objectives c^n differ is that of a technical vs. normative approach. A 
technical approach tries to 'make the setting of objectives "objective," 
and thus preferred! St^osedly, values and norms are -not part of such an 
approach. Of course, however, there are methodological' decisions in any 
technical approach that must be made "non-technical ly.-" 

The existing national averages approach for th^ current standards is 
a technical approach. Used prospectively^, such a system uses the national 
average as the level of attainaient. However, the choice- of a measure of 
central tendency for the level in and of itself is a normative choice. ' 
Why not set the level at the highest agency's level of attainment? 
Moreover, there are myriad measures of central tendency to choose from. 
Why not use the median? These methodological choices will clearly change 
the levels of attainment set. 

■ The- previous BPA proposal for the revised standards is, also a. technical- 

- approecli.- .A clear normative component of this approacfa^however, is asking-, 
ageacies in the low category to improve ISO*,, in the medium category 12S%, 
and in the high category 7S%. Choosing to ask more of the les^ well per- 
forming agencies is clearly a normative choice, • - 
> Onr very technical method considered by BPA is a multivariate predictive 

> method. Such a method would use a time series-cross section data, base of 
attainment data and possible predictors to generate a statistical predic- 
tion of what the level of attainment for a particular agency will be in 
the next time period. The data base woiild include dati.for all VR agencies 
for several years. Possible predictors for a particular data, element woi^ld 
include other data elements and their component parts ^..measures of the 

" emrlronaeht, budge.t figures,. cUent. characteristics, measwre-of- staff-in- _ -- 
put?, measures of services, etc. Of course, as -is true for alL methods 
relyinf- partially or wholly on past attainment, for this system what has • 
been becomes what shall be> ^ 
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Using the normative approach, the objectives would be set by RSA on 
the basis of where the state VR agency "should" go, without recourse to 
technical analysis. Such an approach seems to have been taken in setting 
objectives' for Jhe SSDI/SSI-VR Special Program. , — ^ 

It is clear that all methods for setting objectives will include some 
.mix of the technical and the normative. It is also clear that technical 
.solutions provide no panacea for the difficult problems in* designing a 
method for setting obj,ectives. No simply technical solution is possible. . 

Another very - important distinction is between TniniTmim and m ax iM i m 
obj^tives. It is one thing to say that an agency*^ shoiild rehabilitate at 
le«st 80* of it?, clients at/or above mTTiTmm wage, and another ..that each 
' agency should attain the ideal (e.g; , 100%) . With miniaiums, many agencies 
are likely to ajchieve their objective; with maximums, agencies will likely 
always be striving toward the ideal goal. 

Another way to see this distinction is to look at the differences bet- 
ween "reasonable"- and ''optimal" expectations. With^ minimum. objectives, 
the eaqj'hasis is imially on what is reasonable and feasible-for-an agencjr 
~ although ^^^^^ expectations could b^ set "unreasonabljl'^ at 99* >, for - — 
exaaqjle.' . Optimal e:qpectations imply that the agency operate at what the 
economists call- the frontijw: of the production possibilities^ curve, sqi^eez- 
ing.tixe most out of the agency's resources. 

The question throu^out is which method will lead to better -attaimAent.^ 
The trick would aeea to be to set miniimTa or maxinnms just out of the 
agency's reach. The ag^cy would not find such objectives unreasonable, 
but the agency would also not find the expectation ^ too easy to meet. The 
point is' to' get as much movement out of the agency as possible. 
, Tie- proper setting of this just-out-^tf-reach level -is y^ry- difficult , 
* however;. An analysis of existing data tells, how the agencies are currently* 

operatingV IVhether they arft operating efficiently cannot ^be- determined 
-dfeoir the analysis of aggregate data. Thus, only if one. Is willing to assume 
efficient operation can one use the empirical levels of attainment & gen- 
erate optimal expectations • This, suggests- that the method for setting 
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Objectives will have to be a trial and error one, with "flexibility built 
in, to allow for the "right" level to be obtained through adjustment. 

Logical vs. No Logical Basis 

Some of the data elements have built into them a logical oiniaum or 
maximum. For example, the cost benefit ratio has a logical minimum of 1.0, 
under which costs are greater than benefits. For another exanple, the 
percentage with, "i-p-^^w". wage has a logical maximum of 100* no agency 
can do better. However, in spite of what would" seem a logical mniTm n n or 
maximum; there are difficulties in using. these as expected levels. For 
example, an agency will probably always serve some clients for whom employ- 
ment is not a. goal. It is 100* minus the percentage of such clients which 
is the- logical n^r^"""" — if the validity of serving such clients is allowed. 
Thus, logical rnp-rm,n^ and marlnnm cannot be completely relied upon as a 
method for setting objectives, ^ . 

Historically vs. Ahi&torically Derived ' - 

One-way to set the objectives is to set them on the Basis of the his-- 
'.torical perfoimance o£ the VR agency. One method Which; ii^histprically 
based is described below. . - _- • , . 

A state is judged On the basis of the amount of ia^jrovement it is able 
to m^e from year to year toward achieving the goal. Thi<s, ii a state had 
80V achievement of a. goal one-yea^and 88% the next,, then the attainment 
level vfould be concerned with the rate of the change 0x9X9 10%) or wd^ - 
i^e percent qf the "gap" closed (here, 8/20 or 40%). the level for tMs^ 
type of progress could be specified a number of ways. For example, on\^way 
would be simply to establish a fixed minimum. • All states shall make X% , 
progress from year to year toward achieving the goal. -Alternatively, 
statistical norms could be devised. For instance, "underachievers^J'-ndght _ _ 
Be"those with the lowest rates of progress, or with the least progress in - 
closing the gap. It seems clear th>t the historical pattern for a partial- . . 
lar agency, or for agencies in general, should be included somehow.. in the . 
setting of objectives. , . . ^ 
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National- v's . Sub-National Comparisons 

Both the current method and the previous BPA proposal for the revised 
standards are based in large part; "on national comparisons. The first . 
question, of course,- is whether an agency should be compared to any other 
agencies in the process of setting objectives. Whether or not the answer 
to that question is yes, comparison is useful, even if just for background 
information. 

The second question is whether the comparison should be to all other 
agencies, the national con^arison,. or to some subset of other agencie|| 
The amount of diversity among the 80-odd agencies is staggering, so that 
some subset se«ns useful. Natural subsets ,are available, large vs. small 
agencies, general vs. blind vs. combined agencies, or the- agencies in the 
various federal regions. But, there is diversity there also'. 

A couple of ^preaches ar« possible to identify con^arable agencies-. 
The first is to ask the VR agencies themselves, and to ask RSA personnel, 
what agencies they think are comparable. The questioning could focus on 
one "sister" agency or on. a list of five or more comparable agencies. The 
second is to use existing data on the agencies to empirically. .defiao-similar. 
agencie*, through the family of techniques known as numerical taxonomy. 
" - - Whatever method is used to define comparable agencies, having, a-subset-- 
o£ such agencies seems useful. However, the' very reaL di^ferences-between. 
agencies somewhat limit the role of comparable agencies' attainment in set- 
ting objectives for a particular agency. - 

For Agency vs. For Sub-Agency . ' 

The standards are clearly targeted for the state VR agency. As such, 
the standards are not concerned per se with sub-agency uni^s. Within a- 
state, units such as regions, districts, offices, or even the individual 
counselors cotild be the target of Performance and Procedural Program Stan- 
d«fds. " However, in the federal/state- VR system, these sub-agency ianits . 
»e~not the focus. Thus, in this standards system, the method for setting 
objectives for the agency does not have to deal, with the separate problem 
of specifying objectives for the sub-agency units. State VR agencies are* - 
free to set their own' sub -objectives if desired, and free to choose a 
method for setting these objectives. 

IPS - , N . 
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Adjusted vs. Unadjusted Objectives 

Another possibility is comparability adjustments to .adjust state VR 
agency attainment. There would be adjustments to current attainment, 
adjxasting^, for example, a 70% competitive employment figure up to 80%, 
or down to 63%* However, this s4me level of reasoning could be used to 
adj;i5t objectives, turning a cost-benefit ratio objective of 1,0 into 
1.2S. ^ 

Three needs for ccanpar ability adjus^ents are identified. First, VR 
agencieis differ in factors partially beyond the control of the agency, 
such as unemployment rate, the disabled population, and agency funding • 
Second, the VR agencies differ in client selection, in the kinds and dif* 
ficulty o£ clients that end up being served by the agency. Third, com- 
parability adjustments are necessary to prevent the perverse behavior 
usually 'Msociated with standards systems as different ince^ives and 
disincentives are created. 

In. spite of the need for cdnparability adjxistments, the data and 
methods for making these needed adjustments were found not to be avail- 
able; There is no easily obtainable, agreed-cn comparability adjuiitment. . 
The closest candidate was a client difficulty index, similar to- a: weighted 
case^ closure. For that reason, BPA recommended the inclusion of such an- 



index in the reporting, system, and makes that same recommendation here. 
Single Objectives vs. Incorporating Relationships ^ 



In the Analytic Paradigm, the relationships among the standards, were 
discussed^ That discussion has ' implications for the setting" of objectives. 
The levels of attainiaent could be set independently for each data element, 
without regard to the relationships among data elements. In that wayv^the 
trade-offs involved would not be directly faced. The: problem, however, 
is not now easily resolvable*. The best that can be said- is that these 
IStade-offs will have to be understood in the objective- setting process, 
though the objectives will be set data element by data .element. 
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Stipulated vs. Megotiated Objectives 

A further dimensions o£»methods used to set objectives for state VR 
agency performance is whether RSA will take a top-dowrj stance and once- 
and for all set the objectives for the state ^ agency, or whether the •. 
agency is to participate ia thftTOiiJjective setting process. This is not 
^ to 'say that an agency would set its own objectives. The point is whether 
. the agency will have the chance to respond to the expectations, present- 
ing arguments and evidence of unustial situations or of unknown obstacles. 
in this negotiation process . the ."unreasonableness" of objectives can be 
somewhat resolved. 

. General- V5." Standard Specific Methods 

Implicit ia the discossion of the other dimensions is that the method 
of setting bbj ectives could, be different for each s tand a r d, even for e&ch 
data element. For example, if not enough is known about a particular data / 
element to, provide a clear picture of what good performance should look 
like, then no objective might be set for that data element. Or, for ejcample, 
'u»ier the cost, effectiveness staaiard^ the benefit cost rates could have a 
general minimim objective set an L.O, but the discounted net present value 
could be set by comparison with other agencies of the s'sune size. 




